/jp/ - Otaku Culture

[Anonymous No.]

Ghost Board Synchronization API, Draft 1

This document describes an API for synchronizing local posts ("ghost posts") between archivers.

0. Structure of existing data
Ghost posts are written to both board and board_local tables, and have a subnum greater than 0.
The subnum field is described as an int(10) unsigned.

1. Identification of post source
Beginning from implementation of the API, ghost posts shall be written to the board table with a modified subnum, containing the next subnum on the archiver where the post was made, and the identifier of the archiver where the ghost post originated from.

The archive identifier will be the high two digits of the subnum field, and the local next subnum.
Therefore, a ghost in reply to >>12345,67 on archiver with id 01 will have subnum field 01000068.

The archive id is determined by consensus among archivers and should be unique per archiver.
A suggested id ordering is chronological:
00 jparchive.dyndns.org
01 archive.foolz.us
02 fuuka.warosu.org
03 fuuka.bwaka.net

Since there are no ghost posts with subnums greater than 999999, the presence of non-zero digits in the high two digits may be taken to show that the ghost post was made with an API-compliant archiver.
Archivers may choose to implement the display of API-compliant ghost posts in any manner. For example, a ghost post with num=12345 and subnum=4000026 could be shown to the user as >>12345,26-Short, where Short is a short name corresponding to the archive with id 04.

(continued)

[Anonymous No.]

2. Synchronization API
Archivers shall make available one endpoint for reading its local ghost posts. This endpoint is:
/?task=ghost-sync&since=<UNIX_TIMESTAMP>[&until=<UNIX_TIMESTAMP_UNTIL>]
where <UNIX_TIMESTAMP> is the number of seconds since UTC Jan 1 1970 00:00:00.

The archiver shall return, upon a HTTP GET request to this endpoint, a JSON array of post objects corresponding to ghost posts made locally since <UNIX_TIMESTAMP_SINCE> until <UNIX_TIMESTAMP_UNTIL> unless <UNIX_TIMESTAMP_UNTIL> is not specified, in which case the archiver shall return a JSON array of post objects made locally since <UNIX_TIME_SINCE> until the present.
Post objects should be serialized directly from their MySQL representation, with columns mapping to object keys and row values mapping to their corresponding object values.

If an archiver chooses not to honor a synchronization call, it should respond with a JSON array of a single object with a single key of "ERROR" with corresponding value describing why the call was not honored.

3. Other
This is a draft document, and as such the author welcomes suggestions and alterations to this document.

[Saegrimr !SHIELDM.mE No.]

[Anonymous No.]

>>8101792
Foolz doesn't need lings because no one uses the other retarded archive. Seriously easymodo redirects to foolz. Just use it and stop bitching

[Miles !!dwS94E9FIBt No.]

Why does /jp/ have so many archives?

[Anonymous No.]

>>8101795
If you're going to attention whore for your archive you should leave your name and trip on.

[Anonymous No.]

>>8101807
Get out warosu devs.

[Anonymous No.]

>>8101792
Largest player on the block never goes for shared standards. It's why Microsoft loves its proprietary formats.

[Rozalin !!vPEaWM5chi6 No.]

I'll never understand ghost posting instead of posting on the actual board.

[Saegrimr !SHIELDM.mE No.]

>>8101829
So you can bitch about being banned.

Or talk about /jp/ radio.

[Cudder !MhMRSATORI!!FBeUS42x4uM No.]

Recommendation:
1. Change archive identifiers to GUIDs.
>Since there are no ghost posts with subnums greater than 999999, the presence of non-zero digits in the high two digits may be taken to show that the ghost post was made with an API-compliant archiver.
This is making the assumption that there will never be that many ghost posts. The idiots who wrote non-Y2K-compliant software made the same mistake.

2. I suggest CSV for post data as it is easier and more efficient to parse. Add a ghost-sync-meta request for obtaining the column names and types.

[Anonymous No.]

>>8101839
1. The archive identifiers are int(2) so they can be shoved into the existing subnum field.

Yes, I am making the assumption that there will never be that many ghost posts. Eksopl did too when he made the subnum a int(10) field. I think it's a safe assumption to make, given that the highest subnum so far is >>2538432,1173.

2. A post object keys would provide the data described by such a ghost-sync-meta request, but I suppose CSV is much easier to use. I would support changing the draft to use CSV instead of JSON.

[Anonymous No.]

>>8101839
Ten thousand /jp/ers would need to post a hundred times each in a single thread to get to that point.

I think we're safe.

[Rozalin !!vPEaWM5chi6 No.]

>>8101960
It's better to plan for far higher numbers if you have the ability however.

[Anonymous No.]

>>8101960
You mean ten /jp/ers posting a hundred thousand times in a single thread.

[Anonymous No.]

>>8101839
>web api using csv
Go back to 1998 with your CSV bullshit ``please''

[Anonymous No.]

Just make an extra database field, you retards. This is more stupid that stringifying.

[Anonymous No.]

>>8101812
>NO U
Nice.

[Anonymous No.]

You don't need to sync anything, archives can't be synched effectively anyway. There will be stuff like posts popping out of nowhere, missing posts, replies that quote unsynched posts, etc.

You just need to expose the replies to a particular thread as JSONP. A userscript or a built-in javascript will then fetch the replies from each archive when a thread is displayed, client side.

[Anonymous No.]

Bwaka admin here, Finished improvements for /jp/'s database.

Search should be much faster than it used to be, now has a back-up system, and i'll adding better search features in a while.

On the topic of sharing ghost boards, I suppose it could be done, It doesn't seem too difficult and the ghost databases are alot easier to handle than the larger main databases.

[Anonymous No.]

>>8101792
Only a fool would need an archive other than foolz!

[Anonymous No.]

>>8102734
Typical foolz zealot humour.

[Anonymous No.]

>>8102712
Ghost posts are made to both {board} and {board}_local, and the {board}_local contains incomplete data (only the num, subnum, parent, and timestamp).

Nice work to fuuka. What changes did you make specifically?

[Anonymous No.]

>>8101794
Excellent question.

It's 4 times the usual waste of bandwidth.

[Anonymous No.]

Circlejerking attentionwhores are worse than tripfags from /a/. I'm sticking to Foolz.

[Anonymous No.]

>>8103141
Enjoy your circlejerking attentionwhores then.

[Anonymous No.]

I've always wondered, how do I comment on a Request For Comments? Does it even matter? I mean, it's been published already...

[Anonymous No.]

>>8103083
You can easily fix that though. insert ghost posts into their own table complete and just share thread IDs between the main and local.

That keeps the data independant from the main tables, not sure how that would be performance wise.

[Anonymous No.]

>>8102861
fuck you faggot, I worked all day on that joke.

[Anonymous No.]

>>8102419
This is an interesting way to make it Someone Else's Problem. Maybe a good approach.

[Anonymous No.]

>>8104532
Nothing wrong with offloading as much work as possible to the client.

[Anonymous No.]

>>8104584
Anyone up for writing such a script?

That would reduce the endpoint specification to something like /{board}/thread/{parent}/json/{UNIX_TIMESTAMP}.

[Cudder !MhMRSATORI!!FBeUS42x4uM No.]

>>8102251
I WISH I was still in 1998. Back when the Internet was mostly usable and websites weren't the overbloated AJAX-stuffed-turkey monstrosities they are now.

Posts are stored as relational data. CSV is a standard way of exchanging relational data. There does not need to be anything more complex.

>>8104584
Except when the client is a netbook.

>>8103155
http://www.rfc-editor.org/rfcfaq.html

[Anonymous No.]

>>8105756
Any netbook which can't handle a XHR and a DOM insertion or two should be thrown out a window.

[Anonymous No.]

>>8105291
And you don't need a timestamp, just send every ghost reply for a given thread. It will be easier and more efficient to cache, server side.

[Anonymous No.]

>>8105756
>CSV is a standard way of exchanging relational data for the web.
And you wonder why nobody takes you seriously.

I've yet to see a web API providing data only or primarily in CSV.

[Anonymous No.]

Why does it use a pull model, when a push model would be more efficient and faster, given the small numbers of ghost posts made at any given time?

[Anonymous No.]

>>8106093
>Give random people write access to the database