chrisbrogan.com - Latest Comments in Twitter Needs an Offline Mode and an Open Client

Re: Twitter Needs an Offline Mode and an Open Client

Telemetry — Tue, 27 Mar 2012 20:01:18 -0000

To be honest I cannot say that i full agree on this even now as I do feel its all about speaking as and when you are on there!

Re: Twitter Needs an Offline Mode and an Open Client

youtube downloader — Tue, 07 Dec 2010 07:49:28 -0000

We need one layer of abstraction. One way to do this would be to use a client like Twhirl, and give it the availability to write tweets to two places: an RSS feed (so we could do more with the data- and I want that feature anyway, Loic. Okay?), and the second to an intermediate database somewhere on the Amazon S3 cloud.

Re: Twitter Needs an Offline Mode and an Open Client

chrisbrogan — Sun, 25 May 2008 06:06:34 -0000

@James - some really good points in your reply. You're right that I didn't think much about the block feature, for instance. Same with @replies.

And yet, I think the core of what I'm trying to do is get it to run when the back end is down. It's not that I want to re-architect to clear the flaws. I want the patch to fix the bad stuff.

Still, great points that I wasn't considering much.

Re: Twitter Needs an Offline Mode and an Open Client

James — Sun, 25 May 2008 01:14:31 -0000

Chris,

Your still missing the point of what the problem is. The problem is, from my understanding, in b) Message Processing. The storage is trivial - anyone with a rudimentary understanding of servers can solve that part.

Logically processing the messages in the way twitter is structured is the problem. Now my initial thought is that this suggests they built twitter from the wrong direction when it comes to message processing and that it's a relatively easy fix overall but they started out in the wrong direction architecturally. I could be completely off on this and oversimplifying the architecture and the problem they're facing. But regardless, it's not a hardware problem, it's a software problem either way.

You suggest it's as easy having Twhirl read an XML copy, but think about how many different XML copies it has to make and what decisions have to go into those XML feeds. When Twitter is up and running the API call is simple - Here's a couple of variables, give me the 20 most recent messages along those values. Unfortunately when Twitter isn't up, I don't believe it's anywhere near that simple - at least not given what I think their approach has been so far.

It's not so much that I don't see how your idea would work, it's just that it doesn't factor in a lot about Twitter. For example, Twitter allows you to block people from seeing your updates - that doesn't work if you're throwing up a public feed to everyone regardless (sure there are ways around it but not inherent to twitter itself per se). @ replies are also broken in this model which for many are a crucial function of Twitter.

Ultimately you end up with a band-aid fix that ignores the fundamental problems Twitter is having and simply drags on the problem further. Why not fix part b the Message Processing where the problem actually is and move on instead of backwards ways to pretend to cover up the issue?

Re: Twitter Needs an Offline Mode and an Open Client

Francesco — Sat, 24 May 2008 23:46:52 -0000

Sometimes on the internet we are able to find some ideas that are very useful to us. They can be useful to us in respect of our business or our personal lives. These ideas help us a lot in our decision making process.

Re: Twitter Needs an Offline Mode and an Open Client

chrisbrogan — Sat, 24 May 2008 19:04:42 -0000

Okay, for those who wanted more detail, let me lay it out:

1.) Let's say that Twitter has three primary pieces to its architecture right now (not counting maintenance stuff).
-- a.) Front end interface (that which connects to the web, to SMS, to Jabber, etc)
-- b.) Message gateway - the actual spot where the data gets processed, marked, and stored.
-- c.) Database servers and storage.

2.) In the current situation, part c (servers/storage) goes down, and we're offline.

3.) In my proposal, we do a few things:

-- a.) Add a function to the message gateway to shut down writes to the main Twitter database in times of downtime, and throw a flag to alert that we're on the standby.
-- b.) Add a function to the message gateway to write to a separate database.
-- c.) Add a function to the message gateway *and* a new database to the primary servers to write/store an XML/RSS feed of our twitter stream.
-- d.) Add a function to Twhirl (or an open source Twitter Front End client) that allows Twhirl to detect when Twitter prime is down.
-- e.) Add a function to Twhirl (or similar) that allows Twhirl to write to the secondary database.
-- f.) Add a function to Twhirl to access the XML copy of our stream and the stream of our friends.
-- g.) Add a function to the message gateway that trickle-inserts our "out of service" tweets back into the primary copy of the database.

The outcomes are:

* Twitter functionality runs, even when offline.
* Data retention and integrity.
* Enhanced usability (the RSS feed).

The benefits to Twitter are:

* Continued operation.
* Less bitching and moaning by us.

The benefits to Twhirl are:

* A hands-down reason why we'd use this app as our #1 Twitter interface.

That's the plan, roughly.

Re: Twitter Needs an Offline Mode and an Open Client

Terry Bean — Sat, 24 May 2008 15:46:39 -0000

Chris-

Twitter needs help. I think the idea of tweets having a second home is nice. Maybe somewhere near Maui so we can all go visit.

Ed V. Your message cracks me up.

Re: Twitter Needs an Offline Mode and an Open Client

Carlos R Hernandez — Sat, 24 May 2008 12:26:08 -0000

Chris,

I appreciate the mature and well thought out point-of-view of your post.

I am a blog and Twitter peep of yours ( you were recommended)and now I read why.

:-)

Re: Twitter Needs an Offline Mode and an Open Client

Josh Nichols — Sat, 24 May 2008 09:46:25 -0000

Having a database stored on Amazon S3 doesn't really make much sense. But using Amazon SQS would.

I could imagine a client that does something like:

* When you tweet, it gets added to the queue
* Every so often (whatever the API throttling is capped at), pull a tweet off the queue, and try to post it
* If the post fails, ie twitter is down, make note of it, stick the tweet back on the queue, and wait awhile before trying again

It's moments like this that I wish I knew something about desktop application development.

Re: Twitter Needs an Offline Mode and an Open Client

LisaN — Sat, 24 May 2008 09:23:34 -0000

I'm not too techwise, so most of the post went right over my head, but the part about one tweet getting the mind rolling is what happens to me. One thought generates a thunderstorm of thought, which sometimes, either causes reflection or moves me in a totally different direction than I'd previously thought of.

For me, this is one of the beauties of Twitter, getting new ideas from unlikely places.

BTW, Chris, I tagged you for a meme, http://www.iowaavenue.com/p.... If you're interested, I love to hear your response.................:)

Re: Twitter Needs an Offline Mode and an Open Client

maxinerappleby — Sat, 24 May 2008 08:58:06 -0000

Does anyone know what the relationship is between the outages and new twitter subscribers, downtime to events, or the influx of new twitterfeeds? It would be interesting to see this graph.

Everyone uses twitter for his or her own reason, but for me, I’m a news junkie. I follow many news sources. I notice these outlets use twitterfeed. Sometimes they have so much tweet output that it is necessary for me to choose to unfollow them just to see my friends.

I too have thought about a solution to the Twitter outages and wonder if the Twitter engineers could come up with a tiered level of service for some types of high volume media outlets. As a twitter subscriber, it would be great if I could toggle these groups on/off without having to unfollow each one individually. I think this would also allow the Twitter folks to throttle recourses during peak times much like data centers throttle bandwidth.

Any thoughts?

Re: Twitter Needs an Offline Mode and an Open Client

sachendra Yadav — Sat, 24 May 2008 05:01:08 -0000

Brilliant idea

Re: Twitter Needs an Offline Mode and an Open Client

Ed M — Fri, 23 May 2008 16:05:13 -0000

Joe Cascio will be talking about the idea of "Distributed Twitter" at next Thursday's Boston Ignite Night. So if anyone is in the Boston area next week stop by and discuss.

Re: Twitter Needs an Offline Mode and an Open Client

David Damore — Fri, 23 May 2008 15:45:59 -0000

Great ideas. Your smart readers WILL find and create a workable solution.
BEST!

Re: Twitter Needs an Offline Mode and an Open Client

Geoff — Fri, 23 May 2008 14:44:32 -0000

You can use MessageDance to make tweets when Twitter is down (and they are accessible publicly through RSS). When Twitter comes back, we push them through. We also cache all of your tweets (and those you follow) in our MyTweets section.

Re: Twitter Needs an Offline Mode and an Open Client

Joel — Fri, 23 May 2008 13:59:45 -0000

you're idea is fuzzy and has a lot of potential problems, and to be honest, makes very little sense. Maybe you could give us a more detailed idea of the architecture you envision.

For example, what exactly do you mean by "publishing to an RSS feed." ? Are you seriously suggesting writing tweets directly into a flat text file rather than into a database?

Re: Twitter Needs an Offline Mode and an Open Client

James — Fri, 23 May 2008 13:39:26 -0000

Rick -

Presumably with the $15m they're supposedly closing very soon.

Re: Twitter Needs an Offline Mode and an Open Client

rickt — Fri, 23 May 2008 13:36:50 -0000

Yes, all great comments. And as an enterprise UNIX engineer myself, I agree they are all necessary to ensure Twitter's longterm scalability.

But they are all expensive. Twitter is going to pay for all of this extra hardware, and engineering labs, and $100k+ people to design, test, QA and deploy it how?

Re: Twitter Needs an Offline Mode and an Open Client

James Socol — Fri, 23 May 2008 13:27:29 -0000

You make a good point about Twitter and give me things to think about for my own application design. I've never handled more than 80k requests a day. I'd love to deal with this level of traffic--wouldn't we all--and work on Twitter-sized problems.

Re: Twitter Needs an Offline Mode and an Open Client

Henry Michel — Fri, 23 May 2008 13:27:05 -0000

Awesome suggestion , Chris, though I'm so surprised it doesn't exist yet.

Re: Twitter Needs an Offline Mode and an Open Client

James — Fri, 23 May 2008 13:26:09 -0000

But the problems Twitter is having is in relation to sending out all the updates to all the necessary people. It's not as simple as just dropping it into a database and things magically happening when Twitter itself is down.

You'll miss real time conversation if it's just dropping into an extra database somewhere. And you'll miss the relevancy if you're just storing the updates for later processing by Twitter itself.

Optimization of message handling is ultimately the issue Twitter is having - it's not about throwing a few more boxes at it.

Re: Twitter Needs an Offline Mode and an Open Client

Selina — Fri, 23 May 2008 13:21:03 -0000

Ok Chris, this reminds me of something Len Devanna would say. You two are similar, it is so.

Re: Twitter Needs an Offline Mode and an Open Client

Foomandoonian — Fri, 23 May 2008 13:20:13 -0000

@Edward Vielmetti If you didn't get that comment from http://whentwitterisdown.com/ you should submit it to @lonelysandwich for inclusion! Brilliant!

Re: Twitter Needs an Offline Mode and an Open Client

Ed Vielmetti — Fri, 23 May 2008 13:16:35 -0000

When Twitter is down, I write down messages on a piece of paper until the urge passes. Most of the time the paper can be discarded.

Re: Twitter Needs an Offline Mode and an Open Client

chrisbrogan — Fri, 23 May 2008 13:15:38 -0000

They don't have to quickly process the backlog. They can thread it in over time.

When we're mad about Twitter being down, it's usually because we want the point-in-time conversation. We're at a show, or we're watching the election, or we're looking for people at the Tweetup, or the conference.

The real premise here is that we have something that can handle the crash, keep us typing, and then move us forward until the next opening.