Data @ NewAmerica

Today New America rolled out a microsite showing how -- and where -- attacks on U.S. and NATO troops by Afghan forces have spiked in recent years. The project draws on research done by Jennifer Rowland and Peter Bergen of the National Security Studies Program, and pairs with an op-ed, also published today, on CNN.com.

This was a high-speed project, going from initial conversations to launch in less than 24 hours. That was possible only because the underlying dataset was clean (and small!), and because we could build on some great Afghanistan mapping work done previously by Internews and the National Democratic Institute. Very talented colleagues don't hurt, either.

The one-page site is hosted on Github at http://newamericafoundation.github.com/security/maps/afghanistan.html. Interested in mapping projects like this? Be sure to also check out http://pakistansurvey.org.

The key to insightful data work is, well, insight -- but it's hard to analyze or visualize anything without the right tools. So what follows is a quick roundup of interesting tools, sites and libraries that have popped up recently and are worth a test drive:

DataWrapper

This beta site is the work of ABZV, "a German training institution for newspaper journalists." DataWrapper takes the HighCharts and D3 visualization libraries (New America is a regular user of Highcharts, and we're itching for an opportunity to experiment with D3), and makes it easy for one to simply upload data and start charting -- no jQuery knowledge required.

As the creators note, "data should be the foundation when starting a new story, not an afterthought." They describe Datawrapper as "a tool that makes working from data to story easier."

In many ways, it's similar to the visualizations bundled with Google Charts, but Datawrapper is fully open source -- it can be downloaded and run on any webserver, giving you complete control over both the look-and-feel and the underlying data. Or you can simply upload data to http://datawrapper.de, and spit out slick line, bar, pie and streamgraph charts from there.

Morris.js

For those who aren't afraid of a little javascript. Olly Smith (@olly_smith), has open-sourced his lightweight library for drawing time-series graphs. There's not much here that HighCharts doesn't offer, although HighCharts must be licensed for commercial users. While a not-for-profit organization New America can use that more-robust library without charge, Morris.js is attractive, simple, and free for all.

Browserling

Browser-compatibility testing is not specific to data-driven projects; it's the bane of all website development. (Thanks Internet Explorer...) But with Flash, HTML5, and dozens of different javascript libraries being used for visualizations, being able to see how a project looks across different browsers and versions is key. Browserling.com lets you test across multiple versions of the five major browsers. Your wait in the queue can be rather long (there's a paid service that gets "express lane" service), but it beats running a half-dozen virtual machines of your own to maintain a collection of obsolete software.

Looking for More?

Andy posted here recently about the tools New America uses most often for mapping projects. And we are increasingly using GitHub Pages to prototype and deploy smaller projects. To learn how to build a full-blown site on GitHub's infrastructure, see this post that explains how DevelopmentSeed is doing it.

(Toolbox photo courtesy of Jason Rhode)

Nicolas Garcia Belmonte has mapped 72 hours of U.S. weather data. I don't have anything to add to that, really, other than it's both beautiful and informative, and well worth the look.

As Belmonte explains, "the visualization shows wind direction encoded in line angles, wind speed encoded in line lengths and disk radius, and temperature encoded in hue. All this for about 1200 weather stations across the country."

The underlying data is all from NOAA's National Weather Service. Your tax dollars at work!

This morning, our colleagues with the Committee for a Responsible Federal Budget unveiled their U.S. Budget Watch analysis of the GOP presidential candidates' budget plans. CRFB scored the policy proposals offered by the candidates thus far in the race and estimated what cost these initiatives would add to the debt and deficit. The release has already caught the attention of media covering the race, and the analysis is bound to stir up some response from the campaigns themselves.

We helped CRFB communicate the numbers with Highcharts, creating a slate of line and bar charts suitable for both embedding online and placing into print reports and slideshow presentations. (The ability to easily save off the charts in various formats is one of the reasons we're fond of this charting library. Another is that it's free for non-commercial use!)

The chart below is the interactive overview of the immediate-debt scenarios of all remaining GOP candidates compared to the CRFB realistic baseline of debt as a percentage of GDP:

Feel free to download PDF, PNG, JPG or SVG versions of the above chart for your own use -- just use the download icon in the upper-right corner of the chart. And you can see the rest of the charts over on the Committee for a Responsible Federal Budget website in their report, "Primary Numbers: The GOP Candidates and the Federal Debt."

Mitt Romney's bid for the GOP nomination has brought new attention to the Mormon vote, and the largest home-grown religion's role in public life across the nation. We know that Utah is Mormon Central -- upwards of 80 percent of the population in some counties -- but Liberty County in Florida? Hemphill County, Texas? Who knew?

Now, thanks to our colleagues at Slate, we all do. The map below shows -- in great detail -- where and to what degree Mormons have settled away from Zion.

This is the second Slate/New America Map of the Week. The first -- detailing the global state of mobile-phone penetration -- is available here.

Usually we try to keep the nerd talk to a minimum, but last week we released our first "Map of the Week" with Slate.com, and we got a few questions about the tools we are using to build and present our maps. So what follows is a quick tour of our map stack.

The first thing to point out is that we are using all open source tools. In general, we are fans of using the best tool for the job, and also of pushing open technology whenever possible. In this case, the two worlds overlap very nicely. Second, many of the tools we use are being developed by the amazing people at MapBox. If you are interested in the future of mapping, check them out.

Ok, on to the list...

Quantum GIS

Our desktop GIS system is Quantum GIS. With QGIS we can perform all our necessary spatial data processing, including table joins, spatial joins, attribute manipulation, classification, and data export. QGIS is capable of the basic spatial analysis you would expect from a desktop GIS, and you can extend this functionality through third-party plugins. New America has taken advantage of ESRI's non-profit discounts to purchase an ARCInfo Bundle as well, but to date QGIS gives us all the GIS capability we've needed to produce custom spatial data.

TileMill

TileMill is described as a design studio for maps. In TileMill you can layer, style and create interactive spatial data. TileMill is not a full-fledged GIS, rather, TileMill is designed to style and compose interactive maps by using existing spatial data. Usually we do the bulk of our data processing in QGIS then bring it into TileMill. You can input different data formats into TileMill including shapefiles, ZIP files, CSV files, SQLite, and PostGIS databases.

The styling in TileMill is done through use of the Carto styling language. Carto's syntax is similar to CSS which makes it easy for anyone familiar with web design to pickup. Once your maps are styled you can use TileMill to export them in several different formats, including MBTiles. The MBTiles format is essentially tiled map data stored in a SQLite database. Because your data is tiled you can create complex and data-heavy maps that render quickly in web browsers.

MapBox Hosting

To present our tiled maps we use MapBox hosting. MapBox hosting is built on the open source tilestream application and allows you to publish individual sets of tiles which can then be embedded in any web page. You can use either a basic iframe embed or access tilesets through a JSONP url. The latter option is used with a client side mapping library, such as Wax, to present your maps with custom tools and styling. Using the iframe embed is useful for posting maps to blog posts or simple html pages. MapBox hosting also allows you to create composite maps using multiple layers of tiles. You also get statistics on your tilesets so you can evaluate how often your tiles are accessed and since the statistics are tile specific you can see exactly where users interacting with your maps the most.

Wax

We use Wax with modest maps for our custom map interfaces. This allows us a lot of flexibility in terms of styling and how we present our content. For example, in our Debt in the Eurozone map we customized Wax to build a microsite around our map. We created a full page template and customized the tooltips to show below the map with an interactive chart. For our work with Slate we are using Wax with custom CSS and a bit of JavaScript to create a custom Wax map interface with a Flot chart.

Flot

One of our main charting tools is Flot. Flot allows us to embed interactive and dynamic charts on our map pages. We use Flot in combination with Wax and MapBox to produce charts that interact with our maps, just like the one in our "Map of the Week".

Those are the basics. If you've got questions or comments feel free to drop me a line on Twitter.

In 2013, student loan interest rates are primed to double, thanks to some creative legislating by Congress.

In the 2006 campaign, Democrats promised voters that they'd cut student loan interest rates in half if elected. But because they couldn't do so without adding billions to the deficit, they compromised by slowly lowering the rate and meeting their campaign promise for just one year — after which interest rates snap back up to 6.8 percent.

And while this change will be stark, the political jockeying that produced it is hardly new. To illustrate this, we took data from Jason Delisle's issue brief, "Student Loan Interest Rates: History, Subsidies, Cost," and plotted both the the path and politics of interest rates over the past 20 years.

To do this, we worked with the Highcharts API and added some scripting of our own to provide an expanded "tooltip narrative" when users scrolled over points on the chart. Now, instead of deciphering a dense table of congressional actions that inched the rate to and fro over the years, users have a quick entry point into the crux of each rate shift and the reason Congress acted the way that it did.

Embedded below is the chart, but don't forget to read Jason's brief. It has everything you want to know about the history, subsidies, and actual cost of federal student loans.

Today New America launched a new "Map of the Week" dataviz feature with Slate. Each Thursday, we'll be working together to map some policy or social story of note.

The first edition details the explosion of mobile-phone subscriptions around the globe -- and hints at how those growth rates alone aren't enough to transform development efforts in Africa and elsewhere. (These issues were explored in more detail today at our Global Assets Project/Open Technology Initiative event, "Mobile Disconnect: Can Mobile Solutions Really Combat Global Poverty?").

The basic map is embedded below, but for the complete picture, go to Slate.com.

There are lots of interesting maps in the pipeline for the coming weeks, but we're always on the lookout for additional ideas. So if you have a dataset that deserves attention, please let us know.

Over the past decade, mobile connectivity rates have exploded — the worldwide penetration rate is now as high as 87 percent, according to the International Telecommunications Union. And with this better connected world, we've seen some amazing developments in social and political process thanks to the power of the Internet. Of course the Arab Spring is a great example of this, but as we progress toward connected globe, there's even greater hope for other fundamental changes.

But there remains a staggering digital divide that raises even more questions as to whether we have leveraged our new-found mobile connectivity in the most positive ways. This week at New America, we're exploring issues related to mobile connectivity and how it compares around the globe in a special event we're calling "Mobile Disconnect: Can Mobile Solutions Really Combat Global Poverty?"

New America has mapped the latest country-by-country data on this topic; that snapshot of the huge disparities will be unveiled and discussed at the Feb. 9 event.

When it comes to the mobile phones vs. the old-fashioned, fixed-line variety, the numbers are staggering -- in several African nations the ratio tops 200-to-1. But those numbers don't necessarily translate into truly widespread access -- and they don't begin to explain the other obstacles to transformative mobile development efforts.

There's much more to this story, and we hope you tune in on Thursday. We'll post more here when after the event -- and be sure to follow along on Twitter at #mdisconnect starting at 3:30 p.m. Thursday.

Preview our data: Here's a look at the number of mobile phone subscriptions per 100 residents for all nations. We're working with data provided by the International Telecommunications Union.