So what is Splunk? At its core Splunk is a search engine. It was designed 
to allow any data from an infrastructure device to be indexed and searched. Any output from applications, servers and network devices can be “eaten” by Splunk. However, Splunk has become more than just a standalone product. The current 3.x series of the product has opened up the internal API and exposed it to allow outside development of new applications on top of the Splunk core. This post is going to touch on some of the capabilities available to developers looking to get even more out of their Splunk installation.
I am going to be discussing two elements of Splunk that a user can customize and enhance in the current product release: Spunk UI customization and RESTful applications.
UI Customization
The Splunk web UI would definitely be classified as a Web 2.0 application. It relies heavily on the use of asynchronous Javascript and XML (also known as AJAX) and cascading style sheets (CSS).
Splunk uses a set of CSS and supporting HTML and image files to provide themes to the end user. Creating a new theme for use within Splunk is as simple as cloning one of the existing theme’s CSS and supporting files and editing it. I would not recommend trying to create a new theme from scratch as you may miss one of the necessary elements used in the UI resulting in your Splunk UI appearing to be broken.
The base directory for Splunk’s themes is
$SPLUNK_HOME/share/splunk/search_oxiclean/.
Within that directory are the CSS files,
$SPLUNK_HOME/share/splunk/search_oxiclean/static/css/skins/,
and supporting image files,
$SPLUNK_HOME/share/splunk/search_oxiclean/images/skins/.
To help in creating a new theme I highly recommend using the Firefox plugin Firebug to identify which element in the CSS you need to modify to affect the look of the UI.
A simple example of a customization would be to replace the Splunk logo in the upper left corner of the search area with your organizations logo. To do this clone one of the three default themes and modify the following sections (also don’t forget to put the image files on the Splunk server too):
a#logo {
background-image:url(/images/skins/basic/logo_splunk.png);
background-repeat:no-repeat;
}
a#bigLogo {
background-image:url(/images/skins/basic/logo_big_splunk.png);
background-repeat:no-repeat;
background-position:right 64px;
}
Change the background-image:url(…) to your logo, restart Splunkweb, login and change your preference to your new theme.
Unfortunately, creating a custom theme is the extent of the UI customization supported by the current version (3.x). A new version, 4.0, is scheduled to be released in early 2009 that promises much more flexibility in building a custom UI. All elements of the UI are going to be modular and developers will be able to not only rearrange them within the page layout but also be able to create entirely new modules.
RESTful Applications
Splunk was built using the representational state transfer (REST) architecture. The newer versions (3.3 and on) began exposing Splunk’s internal REST API for developers to build upon. A number of REST endpoints were created that developers can directly access from external applications via simple HTTP requests.
A community driven Google code group has been create called Splunk Labs to enable developers to share ideas and applications built upon Splunk. There are currently SDKs available in most major Web application languages including Python, Perl, PHP, Java and .NET.
Out of the box, Splunk includes a number of endpoints that can be accessed directly with a standard web browser. If you have not modified the management port used by splunkd you can view the REST API directly by browsing to https://localhost:8089/services/.
Here are a two of the more interesting endpoints included:
To monitor the status of existing jobs in the system go to https://localhost:8089/services/search/jobs. From there you can examine what searches are running and either cancel them or pause them (yes, you can actually pause searches!)
To view the current configuration files Splunk is using look at https://localhost:8089/services/properties/file_name. Replace “file_name” with any of the .conf files Splunk uses but do not include the .conf extension. (Ex: to view inputs.conf only use /properties/input).
New endpoints can be created by defining them in the restmap.conf file and then creating a new application under the …/etc/apps/ directory. If you are interested in more of the details on coding a custom endpoint I would recommend visiting the Splunk Labs website.
A final comment about the Splunk REST API. In the upcoming 4.0 release, Splunk has stated that they will be greatly expanding the API.