-
Website
http://www.chrisbrogan.com/ -
Original page
http://www.chrisbrogan.com/twitter-needs-an-offline-mode-and-an-open-client/ -
Subscribe
All Comments -
Community
-
-
Top Commenters
-
Popular Threads
All Comments
awesome post... finally someone asks twitter - where is the queue?
See more here...
Twitter-proxy: Any Interest?
http://assetbar.wordpress.com/2008/02/08/twitte...
The idea of having a redundant system is definitely necessary to handle down times, but it does nothing for the problems they're suffering now unfortunately.
(Barely related sidenote: I miss using Twhirl! Adboe AIR doesn't really work properly on Linux yet. Someone should make the point to them that cross-platform means cross-platform!)
When we're mad about Twitter being down, it's usually because we want the point-in-time conversation. We're at a show, or we're watching the election, or we're looking for people at the Tweetup, or the conference.
The real premise here is that we have something that can handle the crash, keep us typing, and then move us forward until the next opening.
You'll miss real time conversation if it's just dropping into an extra database somewhere. And you'll miss the relevancy if you're just storing the updates for later processing by Twitter itself.
Optimization of message handling is ultimately the issue Twitter is having - it's not about throwing a few more boxes at it.
But they are all expensive. Twitter is going to pay for all of this extra hardware, and engineering labs, and $100k+ people to design, test, QA and deploy it how?
Presumably with the $15m they're supposedly closing very soon.
For example, what exactly do you mean by "publishing to an RSS feed." ? Are you seriously suggesting writing tweets directly into a flat text file rather than into a database?
BEST!
Everyone uses twitter for his or her own reason, but for me, I’m a news junkie. I follow many news sources. I notice these outlets use twitterfeed. Sometimes they have so much tweet output that it is necessary for me to choose to unfollow them just to see my friends.
I too have thought about a solution to the Twitter outages and wonder if the Twitter engineers could come up with a tiered level of service for some types of high volume media outlets. As a twitter subscriber, it would be great if I could toggle these groups on/off without having to unfollow each one individually. I think this would also allow the Twitter folks to throttle recourses during peak times much like data centers throttle bandwidth.
Any thoughts?
For me, this is one of the beauties of Twitter, getting new ideas from unlikely places.
BTW, Chris, I tagged you for a meme, http://www.iowaavenue.com/profiles/blog/show?id.... If you're interested, I love to hear your response.................:)
I could imagine a client that does something like:
* When you tweet, it gets added to the queue
* Every so often (whatever the API throttling is capped at), pull a tweet off the queue, and try to post it
* If the post fails, ie twitter is down, make note of it, stick the tweet back on the queue, and wait awhile before trying again
It's moments like this that I wish I knew something about desktop application development.
I appreciate the mature and well thought out point-of-view of your post.
I am a blog and Twitter peep of yours ( you were recommended)and now I read why.
:-)
Twitter needs help. I think the idea of tweets having a second home is nice. Maybe somewhere near Maui so we can all go visit.
Ed V. Your message cracks me up.
1.) Let's say that Twitter has three primary pieces to its architecture right now (not counting maintenance stuff).
-- a.) Front end interface (that which connects to the web, to SMS, to Jabber, etc)
-- b.) Message gateway - the actual spot where the data gets processed, marked, and stored.
-- c.) Database servers and storage.
2.) In the current situation, part c (servers/storage) goes down, and we're offline.
3.) In my proposal, we do a few things:
-- a.) Add a function to the message gateway to shut down writes to the main Twitter database in times of downtime, and throw a flag to alert that we're on the standby.
-- b.) Add a function to the message gateway to write to a separate database.
-- c.) Add a function to the message gateway *and* a new database to the primary servers to write/store an XML/RSS feed of our twitter stream.
-- d.) Add a function to Twhirl (or an open source Twitter Front End client) that allows Twhirl to detect when Twitter prime is down.
-- e.) Add a function to Twhirl (or similar) that allows Twhirl to write to the secondary database.
-- f.) Add a function to Twhirl to access the XML copy of our stream and the stream of our friends.
-- g.) Add a function to the message gateway that trickle-inserts our "out of service" tweets back into the primary copy of the database.
The outcomes are:
* Twitter functionality runs, even when offline.
* Data retention and integrity.
* Enhanced usability (the RSS feed).
The benefits to Twitter are:
* Continued operation.
* Less bitching and moaning by us.
The benefits to Twhirl are:
* A hands-down reason why we'd use this app as our #1 Twitter interface.
That's the plan, roughly.
Your still missing the point of what the problem is. The problem is, from my understanding, in b) Message Processing. The storage is trivial - anyone with a rudimentary understanding of servers can solve that part.
Logically processing the messages in the way twitter is structured is the problem. Now my initial thought is that this suggests they built twitter from the wrong direction when it comes to message processing and that it's a relatively easy fix overall but they started out in the wrong direction architecturally. I could be completely off on this and oversimplifying the architecture and the problem they're facing. But regardless, it's not a hardware problem, it's a software problem either way.
You suggest it's as easy having Twhirl read an XML copy, but think about how many different XML copies it has to make and what decisions have to go into those XML feeds. When Twitter is up and running the API call is simple - Here's a couple of variables, give me the 20 most recent messages along those values. Unfortunately when Twitter isn't up, I don't believe it's anywhere near that simple - at least not given what I think their approach has been so far.
It's not so much that I don't see how your idea would work, it's just that it doesn't factor in a lot about Twitter. For example, Twitter allows you to block people from seeing your updates - that doesn't work if you're throwing up a public feed to everyone regardless (sure there are ways around it but not inherent to twitter itself per se). @ replies are also broken in this model which for many are a crucial function of Twitter.
Ultimately you end up with a band-aid fix that ignores the fundamental problems Twitter is having and simply drags on the problem further. Why not fix part b the Message Processing where the problem actually is and move on instead of backwards ways to pretend to cover up the issue?
And yet, I think the core of what I'm trying to do is get it to run when the back end is down. It's not that I want to re-architect to clear the flaws. I want the patch to fix the bad stuff.
Still, great points that I wasn't considering much.