OSB data in simple text format

I merged and tweaked a couple of example programs from the open source "shapelib" project for reading ESRI shape files, and Anthony ran the program over the OSB data to produce the text file in
http://www.cs.uoregon.edu/classes/08F/cis422/data/OSBShpExtract.txt

This is not in xml, but it's a very simple format that can be parsed with either a scripting language like Python, Perl, or Awk, or with Java's "tokenize" method --- each line can be identified by the first token on the line, and the important fields (x and y coordinates of points, in particular) are separated by blanks.

Of course no one is obligated to do anything with this data at such a late date, but there it is if you want to take a shot at it. Later (probably during holiday break) I'll see about making a version of the extractor program that produces XML in a form close enough to the campus map XML input files to "fool" the data readers you have produced.

Data, again

Many thanks to Daniel, who apparently knows the magical incantations necessary to transform ESRI shapefile data into XML files. If you look in
http://www.cs.uoregon.edu/classes/08F/cis422/data/OSBxml/
you will find the full collection as osb_gis_xml.tar.zip or you can look in the osb_gis_xml directory to get individual xml files.

It's very late, and I understand if you are not able to incorporate this into your project before the deadline. On the other hand, it would be great if at least some of you could do so, or at least give it a try and document what problems need to be solved to make it work.

Update: I've looked through the XML data and I'm not so sure we're making progress. As near as I can tell, what we have is "PolygonB" objects that are represented as byte arrays (and I don't know how to interpret those byte arrays). It seems that "PolygonN" objects have a field called "Rings", which are arrays of (x,y) coordinate pairs, and that's what we were expecting. I don't know how to turn "PolygonB" objects into "PolygonN" objects. Maybe someone does?

OSB data? Please have a look

I have an XML file from Amy. It's one big XML file, rather than a file per layer, but maybe that's not a problem (??). As I eyeball it, though, it seems to me like the shapes are defined in a different way than in the campus map XML files ... it looks like there is a sort of hex encoding of sequences of coordinates, instead of XML-ish encoding of each coordinate pair. So, my best guess is that this will not work with the "generalized" input modules that you have worked on. But please give it a try, and let me know what you discover. Is there some obvious way to transform this XML into something we can deal with?

I also have the raw ESRI shapefile data, if anyone knows the ArcGIS program well enough to manage the XML export from that. (I don't, but I am beginning to see that it's something I had better learn.)

The XML file is at: http://www.cs.uoregon.edu/classes/08F/cis422/data/OSBdata.xml

Some test data, but ...

Amy exported some XML from a City of Eugene geodatabase for me, and I've put it in an accessible place, but I'm not sure how useful it will be. It looks like the individual layer files are basically empty of actual shape data. The full XML dump of the database is quite large (53M), but I'm not sure if it contains data that corresponds to layer data in the campus map. If you want to give it a try, it's at
/home/faculty/michal/public_html/08F-GIS-Eugene
on the Sun (ix) file system, or
http://www.cs.uoregon.edu/~michal/08F-GIS-Eugene
through the web server.

The .shx files are in a native ESRI data format, and the .dbf files are some kind of native database file ... the Unix "file" command claims they are in DBase 3 data file format, which is possible. The "file" command claims the .mdb file is a Microsoft Access database, but that seems unlikely. If there is data useful for testing, it is likely to be in the file XMLExport_GeoDataBase.xml . Warning, that's 53 megabytes of XML text with really long lines. Whether it has anything usable or not, I really can't tell ... trying to look through it with Emacs didn't get me very far, and it crashed Firefox!

We're hoping for real OSB data at the end of the week.

OSB data new ETA - next week

I got word from Amy that there were some problems getting the OSB data into the geographic database. The problems have been worked out, but the new projected time to get the data is next week. That's awfully late in the (first half) project, and I worry that you won't have time to deal with whatever surprises turn up when you're working with a new data set (and I can almost guarantee there will be some surprises, though I don't know what they will be).

On the positive side, this is a realistic experience ... dealing with some external dependence that was not resolved as planned happens all the time.

So how do we cope with it? Here are my thoughts:

• It's worth at least taking a shot at handling the OSB data when we get it ... but I won't be surprised if the result of that is identifying some problems rather than solving them. Identifying problems is progress.
  
• I've asked Amy if we can get some different GIS data, possibly from downtown Eugene, so that we can have some kind of test of the ways you are generalizing the input handling.
  
• If all we can do is test the generalized input processing on the existing input data ... it's not what I hoped for by the midterm, but it's better than nothing.
  

So we go forward as best we can.