Feature of the week: person name normalization in "Last, First" form

As announced earlier, we changed the format of person names for author and editor fields in publication posts from First Last to Last, First in the last release. Since this has quite some implications, I would like to discuss the changes a bit more in detail in this feature of the week.

Why "Last, First"?

The change from the First Last format (e.g., "D.E. Knuth") to Last, First ("Knuth, D.E.") was an overdue step that has been requested by many users of BibSonomy (see also the comments on our blog post).

The old format did not allow our users to correctly store user names that contain two last names. For example, in the name of our colleague Beate Navarro Bullock the first name was erroneously detected to be "Beate Navarro" but it really is only "Beate". When such data was exported from BibSonomy, other systems could not repair it. For instance, in a literature list this would have caused a wrong position of the reference (under "B" instead of "N" in our example). Furthermore, name based citations like [Navarro Bullock et al., 2009] would have used the wrong last name ("Bullock" instead of "Navarro Bullock").

With the new Last, First format (Navarro Bullock, Beate), the name is correctly stored and recognized by other applications and BibSonomy itself (e.g., for the author pages). BibSonomy does no longer "destroy" correctly entered names!

By the way, the new format also correctly handles the lineage (e.g., "Jr.") of persons. You can enter it in the form Last, Jr, First and you will get it back in the same format.

Interaction with BibSonomy

Naturally, this change is of big importance to all users and applications that import data into BibSonomy or export their data from BibSonomy. Since BibSonomy supports a wide variety of import and export formats, I will here briefly explain how each format is handled.

• In general, each input format supports both First Last and Last, First. E.g., XML and JSON with BibSonomy's REST-API and the BibTeX and EndNote import in the web interface.
• When editing a publication reference, you can basically use the format you like. However, after saving the post, the names will be normalized and when you edit it the next time, they will be in Last, First
  order:
  
  
  This has the advantage that you can see if both parts of the name were correctly identified.
  
• Both the XML (format=xml) and the JSON (format=json) output of BibSonomy's REST-API use Last, First for the author and editor attributes of the bibtex element.
• Nothing has changed in BibSonomy's "regular" JSON export. Person names are returned in "First Last" form:
  

  "author": [ 
      "Douglas Crockford"
    ],
  

  After the next release (scheduled for October 26th), the JSON output will additionally include the fields "authors" and "editors" (notice the "s" at the end) with separated "first" and "last" parts:
  

  "authors": [ 
      {"first" : "Douglas", "last" : "Crockford"}
    ],
  

• The BibTeX export now returns authors and editors in Last, First form. You can change this to First Last by adding the parameter firstLastNames=on to the URL. Alternatively, you can use our new export dialog that is triggered by moving the mouse over the BibTeX export link on each page:
  
  
• The EndNote, RIS (EndNote), and RIS (ReferenceManager) export all use the Last, First format.
• For all other export formats we did not change the person name format (to the best of my knowledge).
  

Please note that most applications (like BibTeX, JabRef, Citavi, or EndNote) support both types of formats anyway and thus Last, First should not bring you any problems but rather the possibility to correctly represent more types of person names than First Last.

What else did change?

During our intensive tests of the new format we realized that BibSonomy has quite a lot of data that is - to be hones - broken, dirty, inconsistent. E.g., author fields like "A. Einstein and" or "Knuth, D.E., Kleinberg, J.", etc. In short: strings where often there is no hope that we can automatically and correctly clean them.
What was more important, however, was the fact that due to the new normalization some hashes of posts, i.e., their unique identifiers changed. When we realized this we thought about the implications and finally looked at the numbers - of 2795609 posts only 20215 posts (less than one percent) changed. And as said - almost all of these posts had broken or "dirty" person names and clearly stemmed from broken batch imports.

Feature of the week: access your posts in Emacs (with RefTeX)

This week's feature of the week is something for our more experienced users but I guess that applies to many Emacs users anyway. ;-)

For all of you that write their LaTeX files using GNU Emacs I present a small script that simplifies its interaction with BibSonomy.
First, if you don't know and use RefTeX yet, I highly recommend that you give it a try, it's really awesome! I won't go into detail how to activate it, typically something like

; load reftex                                                                                                
(require 'reftex)
; turn reftex on                                                                                             
(add-hook 'LaTeX-mode-hook 'turn-on-reftex)   ; with AUCTeX LaTeX mode

in your Emacs configuration file ~/.emacs should be sufficient.
Adding citations to your LaTeX file with RefTeX is as easy as typing C-c [. Then you can search for appropriate entries using a regular expression.

What I want to show you in this feature of the week is how you can add a script to RefTeX such that each time RefTeX is initialized your BibTeX references are downloaded from BibSonomy. For this to work you must follow the following steps:

1. Find out which posts you want to use from BibSonomy.
2. Add an appropriate \bibliography{} command to your LaTeX file.
3. Write a small shell script.
4. Tell RefTeX about the script.
5. Try it!

Find out which posts you want to use from BibSonomy

Let us assume that I am writing a paper about formal concept analysis (fca). So I want to reference publications from my list of fca-related publications that I can get at http://www.bibsonomy.org/user/jaeschke/fca. Since I want all publications in BibTeX format, the file I finally want to have is http://www.bibsonomy.org/bib/user/jaeschke/fca?items=1000. Notice the modifications in bold print.

Add an appropriate \bibliography{} command to your LaTeX file

Since the script presented later is rather simple, this is related to the posts I want to download, i.e., to the tag fca:

\bibliographystyle{plain}
\bibliography{fca}

That means the BibTeX file will be named after the tag(s) used to query BibSonomy.

Write a small shell script

Of course, we have done this for you. :-) Nevertheless, you should adopt it to your needs, in particular change the variable BASEURL (e.g., insert your BibSonomy username):

#!/bin/sh

FILE=$1
RESULTFILE=`pwd`/$FILE.bib
BASEURL=http://www.bibsonomy.org/bib/user/jaeschke
PARAMS="?items=1000"
QUERYURL=$BASEURL/$FILE$PARAMS

# the file is downloaded on each call of the script - 
# typically on the first call of reftex-citation
wget -q -O - $QUERYURL > $RESULTFILE

# the result for RefTeX: the path to the downloaded file
echo $RESULTFILE

The script gets as input from RefTeX the string from the \bibliography{} command (if you have several files specified, the script is called for each file). Then, it downloads your posts from BibSonomy using the input as tag of the user jaeschke. The downloaded file is put into the current directory (named after the tag(s)) and its name is returned to standard output (this is what RefTeX expects).

Place the script at some convenient location, let's say /usr/local/bin/find_file_from_bibsonomy.sh.

Of course, the script can and should be adopted to your needs. If you have suggestions for improvements, just let us know!

Tell RefTeX about the script

Add the following lines to your ~/.emacs file:

; connect RefTeX with BibSonomy                                                                        
(setq reftex-use-external-file-finders "yes")
           
(setq reftex-external-file-finders 
   (cons '("bib" . "/usr/local/bin/find_file_from_bibsonomy.sh %f") 
      reftex-external-file-finders)
   )

Try it!

Well, just open the LaTeX file and press C-c [ on your keyboard (for non-Emacs users: Pressing the Ctrl-Key together with "c" and then releasing both keys and pressing "["). Then search for some author name or keyword (e.g., "fca")

and be surprised about the results

You can choose any of the matching posts and the corresponding \cite{} command is automatically added to your document. And the best thing is: all the posts are coming from BibSonomy!

Further reading

• RefTeX in a Nutshell
• RefTeX options for finding files

Improvements

The script above always tries to download the entries from BibSonomy - even if you already have one with that name. Here is an updated version that acts only if the requested file name starts with a configurable prefix, in this case bibsonomy_.

#!/bin/sh                                                                           

# All files with publications from BibSonomy should have
# this prefix - to not overwrite other files.
PREFIX="bibsonomy_"
# query                               
BASEURL="http://www.bibsonomy.org/bib/user/jaeschke"
PARAMS="?items=1000"

# the first parameter is the file name from the \bibliography{}
# command                                  
FILE=$1

# check for prefix
if [ $(echo $FILE | grep "^$PREFIX") ]; then
    # prefix found -> work on

    # remove prefix for tag query
    TAG=$(echo $FILE | sed "s/^$PREFIX//")
    # build path for result file and query URL
    RESULTFILE=`pwd`/$FILE.bib
    QUERYURL=$BASEURL/$TAG$PARAMS

    # the file is downloaded on each call of the script - typically on
    # the first call of reftex-citation
    wget -q -O - $QUERYURL > $RESULTFILE

    # the result for RefTeX: the path to the downloaded file
    echo $RESULTFILE

else
    # just return the file name in case the file is in the current
    # directory                                
    echo $FILE.bib
fi

That means that in the example above we must change in our LaTeX document \bibliography{fca} to \bibliography{bibsonomy_fca}. That's it!

Further improvements will be added to this section.

(Classic) Feature of the week: Posting Publications as snippet

Hello everyone,

this week we learn how to post a publication using a snippet. This is a very comfortable way to post a publication. Just click on post publication and choose the tab snippet.

We can paste one or multiple snippets into the BibTeX/EndNote snippet* box. The visibility of our post is controlled with the viewable for field.

Basically, that is all we have to do. But our snippet importer offers some additional goodies. We can choose to edit before import or to overwrite an existing one.

The best part is the tag capability of BibSonomy. Many snippets already have keyword or tag field as in the following example:

We can use these tags for our import. Since there is no standard for tags and their delimeters, we have to tell BibSonomy our format. Therefore, we have to select the type of tag delimiter from the corresponding drop down menu. We can choose between:

• ' ' (whitespace),
• , (somma), and
  
• ; (semicolon).

We have to set a whitespace substitute sign if we do not use whitespaces as tag telimiter. A '_' is used otherwise. To specifiy our own one, just klick into the text field and remove the underscore and replace it with our own sign.

After we click on the post button appears the next dialog:

It is just a confirmation dialog to be sure that our tags get imported correctly. If everything is fine, we can click on the update button to finish the import.

That is it for this week. Stay tuned and happy tagging!

Jürgen

Feature of the Week: Export Bookmarks to "electronic"-BibTeX

In our previous post we have discussed six new BibTeX entry types that have been implemented in BibSonomy. This time we will focus on one of those types: “electronic”. The type "electronic" allows you to store references to resources on the web as BibTeX.

Of course the more natural (and comfortable) way of storing references to such resources in BibSonomy is using bookmarks. To make those bookmarked references available as BibTeX we have included a BibTeX export for bookmarks:

Simply add “/bookbib” to any BibSonomy URL to get BibTeX entries of all bookmarked resources of the page. For example while http://www.bibsonomy.org/tag/puma shows you a collection of bookmarks and publications with the tag “puma” http://www.bibsonomy.org/bookbib/tag/puma
will give you all BibTeX entries to all the bookmarked resources of said collection.

Here is an example of such a bookmark exported to BibTeX:

@electronic{puma.uni-kassel.de,
title = {PUMA :: home},
url = {http://puma.uni-kassel.de/},
biburl = {http://www.bibsonomy.org/url/e51e87299fc3778064426991cb36633d/sdo},
keywords = {PublicationManagement kassel puma webService},
added-at = {2010-09-03T15:35:58.000+0200},
description = {Puma erleichtert Ihre tägliche Arbeit mit Publikationen und Lesezeichen},
interhash = {e51e87299fc3778064426991cb36633d},
intrahash = {e51e87299fc3778064426991cb36633d}
}

The referenced URL is naturally stored in the “url” field of the entry. The URL's host serves as BibTeX key. The fields “title” and “description” are determined by the title and description of the corresponding BibSonomy bookmark post. As usual “keywords” holds the post's tags, “added-at“ tells you when the the post was added (or last modified) and “biburl” is the link to the bookmark-post.

We are looking forward to any suggestions of further BibTeX fields that could be included in the export.

Feature of the Week: New BibTeX Entry types

When a publication is posted to BibSonomy one of the most important fields of that post is the publication's entry type. There are 14 regular BibTeX entry types - e. g. “article” or “book” being the most common ones.

BibSonomy has supported 13 of those types and in our last release we have added the missing 14th: “conference”.

Over time, there has been great demand for a new BibTeX entry type that allows posting presentations. We are happy to announce that “presentation” is one of six new non-regular (!) entry types, that we have implemented in BibSonomy. Here is a list of the new types:

• electronic: references electronic publications like articles on the web or blog posts
• patent: references patent documents
• periodical: references magazines or other regularly appearing publications
• preamble: references preambles
• presentation: references slides from your talks, seminars or lectures
• standard: references documents describing norms, guidelines, etc.

The new types can be selected during the publication posting.
Our system tag “sys:entrytype:” can help you search for publication posts of a desired type. E. g. searching for “sys:entrytype:standard” returns only publications of type “standard”.
One final remark before you start posting your new BibTeX type posts: Those six new types are non-regular types. That means, they are used in BibSonomy and other applications but they do not belong the the 14 canonical types that are usually understood by all BibTeX styles. Depending on what style you use for your literature references you might sometimes have to adjust either the entry type of the reference or the BibTeX style.

Feature of the Week

Extend your publication entries by adding a .pdf/ps file and a private note

Besides assigning tags to your BibTex entry with BibSonomy, you can extend your entries by uploading a local copy of publication as .pdf or .ps document. In addition, you can assign some private notes, which are only visible for you. This notes can contain some personal thoughts like “this paper is important for my research” or a short summary about the content. For example: You just posted the following publication: “Network Properties of Folksonomies” and would like to extend your entry with a local copy as .pdf or .ps file. You just simply click on the title and get the following options (see figure at the top). Try it and have fun, Miranda!

Feature of the Week: Keep and collect interesting publications in your basket

For collecting publications from other users, BibSonomy offers a basket function. To add entries to your basket, a pick function has been attached to each BibTeX entry. If you are interested in a specific BibTeX entry, you can choose the pick option and the entries are added to your basket.

Your basket can be displayed with the basket button at the right menu. On the following page, you can select or remove items from the basket, and export the final list as BibTeX and EndNote format. Additionally, you can update the tags of your BibTeX entries in a batch mode. This option disregards BibTeX entries, which are not your own to avoid changing tags from foreign BibTeX entries.

Try our basket option and have fun, Miranda!

Feature of the Week: Customizable Publication Exports

Many of you may already be using the export facilities for publication entries that BibSonomy offers; until recently, there were options to export your BibTeX as RSS, RDF, Endnote, or one particular plain HTML format.

Now we've revamped the export facilities by including the export layout engine from the great JabRef stand-alone BibTeX manager. Clicking on the more button next to the publication column header will take you to a list of publication output formats that are now available—right now there are 17! These include an RTF export for standard text processors such as Word, CSV which is easily imported to spreadsheets, as well as fancy HTML tables with sorting capabilities that you can integrate right into you web site.

Here are some examples:

• CSV for spreadsheets
• Fancy HTML table
  
• RTF for Word and other text processors
  
• DocBook XML

If there is still a layout missing that you need for your particular application, you can make your own! The JabRef documentation describes how custom export filters can be written using a special markup language. Once you've written your filter, you can upload three files: the piece of markup for formatting each entry, plus a custom header and footer; thus, you can get a complete, valid HTML page with your custom publication list out of BibSonomy which can directly be linked to without further processing. The URL scheme for these layouts is as follows: http://www.bibsonomy.org/layout/[filter]/[rest of BibSonomy URL] e.g. if I want to format my own publications of 2006 with the html filter: http://www.bibsonomy.org/layout/html/user/schmitz/myown+2006 See also Mark Schenk's page for some great examples of JabRef layouts.

Have fun!

Christoph

Feature of the week

Today I want to start a series of blog posts which explain features of BibSonomy which may be not so well known or understood but nevertheless be helpful for our users.

I'll start with a feature we added last week and which enhances the possibilities when uploading BibTeX files to BibSonomy. Often your BibTeX entries already contain tags inside keywords or tags fields. BibSonomy then adds these tags to your entries. Unfortunately, sometimes the tags in these fields are not separated by whitespace but by comma or semicolon. Hence we implemented the option to use another delimiter than whitespace as tag separator when uploading a BibTeX file.

You can find this option below the file upload form on the post bibtex page. Next to the upload button you can find now an options link. When you click on it the following options appear:

• character encoding
• viewable for
• tag delimiter

with the latter being the one I'll explain here. There are essentially three things you can change for this option:

• The checkbox allows you to enable or disable this feature. By default this option is disabled, which means that tags contained in the BibTeX fields tags or keywords are assumed to be separated by white space. If you enable this checkbox, you can configure with the dropdown box the character which separates the tags.
  
• The dropdown box allows you to choose between comma and semicolon as separator. Picking the comma, for example, means that the string contained in the keywords or tags fields of the file's BibTeX entries will be cut into several strings at each occurrence of a comma. This results in several tags which still may contain whitespace because the string "computer algebra, math" would be cut into "computer algebra" and "math". Since BibSonomy does not allow whitespace within tags we have to take care of the space in "computer algebra". Leaving it as is would result in the three tags "computer", "algebra" and "math" -- which may be appropriate or not, depending on the users choice. But the next options allows us to control this behaviour.
  
• The last input box allows to specify a character which will be used to join several words of one tag into a one word tag. The default is "_" resulting in "computer_algebra" for the preceding example. Removing this character completely would result in "computeralgebra" and entering a space would give us two (!) tags "computer" and "algebra". Hence this little input box gives you some power regarding what your tags will look like.
  

I want to finish with the remark that this option may look a little bit complicated which results from the power it has. Therefore I propose to use it first only on small examples to understand how it works. If you have questions or comments, don't hesitate to comment in this blog or contact us.

Robert