Goldin++

Artifactory’s REST API is something I was using quite a lot recently and would like to share my experience here. Normally, there’s no need to turn to the REST API when working with Artifactory and in most cases Maven, Hudson or TeamCity plugins are the only ones “talking” to it.

But in this specific case Artifactory was used as a general storage for company’s binaries (a capability I praised a lot in the past): Java agent downloads packed Lucene indexes from Artifactory where they are uploaded to by a Hudson job running Maven and Spring Batch.

It is an interesting and very educating project which I’ll surely explore in more details later. Today, I would like to talk about the REST part of it, the way it is used with a SpringTemplate and some Artifactory-specific nuances, marked with

1. Choosing the client.

Initially I was considering Jersey as a client library knowing JFrog teams uses it to provide Artifactory’s REST support. But then I decided to stick to Spring since version 3 was already used throughout the project and one of its new features is exactly what I needed here – a RestTemplate. Why not give it a try?

2. DAO.

Accessing any external resource is normally done through a DAO layer. Accessing Artifactory shouldn’t be an exception to this rule so we have a DAO interface and an ArtifactoryDAO implementation.

public interface DAO
{
    int  getLatestVersion ( String serviceName, String dataType );
    File download ( File dir, String serviceName, String dataType, int version );
    void deleteOldVersions( String serviceName, String dataType, int oldVersions );
}

serviceName and dataType are our analogies to Maven’s groupId and artifactId. The interface allows a client to get a number of the latest version, download any version locally and delete old versions.

3. Spring time!

Java client gets a DAO implementation injected to it, which in turn gets a RestTemplate injected as well, configured with a custom message converter, more on that below.

<bean ... >
    <constructor-arg name = "dao"  ref = "artifactoryDAO"/>
</bean>

<bean name="artifactoryDAO" class="...">

    ...

    <constructor-arg name  = "rest">
        <bean class="org.springframework.web.client.RestTemplate">
            <property name="messageConverters">
                <list>
                    <bean class="JsonMessageConverter"/>
                </list>
            </property>
        </bean>
    </constructor>
</bean>

4. Working with a RestTemplate.

As one would expect, working with a RestTemplate is pretty straightforward. From all convenient methods available I picked up an exchange() one to provide a general-usage wrapper:

<T> T request( String url, HttpMethod method, Class<T> responseType )
{
    HttpHeaders headers = new HttpHeaders();
    headers.set( "Accept",        "application/json" );
    headers.set( "Authorization", auth());

    T response = rest().exchange( url,
                                  method,
                                  new HttpEntity<String>( headers ),
                                  responseType ).getBody();

    assert (( response != null ) &&
            ( responseType.isAssignableFrom( response.getClass())));

    return response;
}

There is a need to specify a correct "Accept" header since RestTemplate sends "Accept: text/plain" by default which is not accepted by Artifactory.

With built-in HttpMessageConverter and its sub-classes there’s no need to worry about conversion of a REST response, JSON in our case, to an object:

Map<String, Object> m =
   request( "http://artifactory/api/storage/repo/serviceName/dataType",
            HttpMethod.GET,
            Map.class );

Last request() argument defines a type of an object to which response will be converted, considering its "Content-Type" and converters available. I was mostly using a Map when reading Artifactory responses, like “folder info”.

There is a need to extend MappingJacksonHttpMessageConverter in order to recognize "application/vnd.org.jfrog.artifactory.storage.folderinfo+json", returned by Artifactory as JSON:

import org.springframework.http.MediaType;
import org.springframework.http.converter.json.MappingJacksonHttpMessageConverter;

public class JsonMessageConverter extends MappingJacksonHttpMessageConverter
{
    @Override
    public boolean canRead ( Class<?> clazz, MediaType mediaType )
    {
        return
            ( super.canRead( clazz, mediaType ) ||
              (( "application".equals( mediaType.getType())) &&
              ( mediaType.getSubtype().startsWith( "vnd.org.jfrog.artifactory" ))));
    }
}

5. Reading InputStream and downloading files.

Message converters work really nice if response is not large and fits into memory. But how about downloading files? I can not assume their content will fit into memory and would like to access a raw InputStream.

It is also possible but in a different way, using an execute() method with two callbacks:

import org.apache.commons.io.IOUtils;

...

private static final RequestCallback ACCEPT_CALLBACK =
    new RequestCallback()
    {
        @Override
        public void doWithRequest ( ClientHttpRequest request ) throws IOException
        {
            request.getHeaders().set( "Accept", "application/json" );
        }
    };

private static class FileResponseExtractor implements ResponseExtractor<Object>
{
    private final File file;
    private       File file () { return this.file; }

    private FileResponseExtractor ( File file )
    {
        this.file = file;
    }

    @Override
    public Object extractData ( ClientHttpResponse response ) throws IOException
    {
        InputStream  is = response.getBody();
        OutputStream os = new BufferedOutputStream( new FileOutputStream( file()));

        IOUtils.copyLarge( is, os );
        IOUtils.closeQuietly( is );
        IOUtils.closeQuietly( os );

        return null;
    }
}

...

rest().execute( downloadUrl,
                HttpMethod.GET,
                ACCEPT_CALLBACK,
                new FileResponseExtractor( new File( downloadDir, fileName )));

Note that to download an artifact with a REST "/api/download/" method requires Artifactory Power Pack license but you can also download an artifact with a regular URL like
"http://artifactory/repo/serviceName/dataType/version/dataType-version.tar".

6. Downloading large files.

I blogged recently about ServletResponse.setContentLength() problem: it doesn’t work with files larger than 2Gb and sends a negative "Content-Length" header due to int overflow.

This problem has also appeared in Artifactory bit it is fixed already, we just need to wait for version "2.2.6" to be released.

Spring 3 uses Apache HttpClient downloading an empty file if negative "Content-Length" header is sent. So I could not download files larger than 2Gb with Spring but I could with wget!

import org.apache.commons.exec.CommandLine;
import org.apache.commons.exec.DefaultExecutor;

...

String command  = String.format(
    "wget -S -nv -O \"%s\" -T 120 \"%s\" --cache=off %s",
    destinationFile, sourceFileUrl, credentials );

int    exitCode = new DefaultExecutor().execute( CommandLine.parse( command ));

As it appears, "wget" works just fine with a negative "Content-Length" which, btw, is not that unusual and can be sent in those cases when response length is not known in advance.

7. Deleting artifacts.

Artifactory’s API doesn’t really mention it but artifacts or folders can be deleted as well, provided client has enough permissions to do so, of course:

request( "http://artifactory/repo/serviceName/dataType",
         HttpMethod.DELETE,
         Map.class );

8. Security.

HTTP calls can be authenticated by overriding Spring’s CommonsClientHttpRequestFactory and configuring RestTemplate to use it:

public class ArtifactoryCommonsClientHttpRequestFactory
    extends  CommonsClientHttpRequestFactory
{
    private final String user;
    private final String password;

    public ArtifactoryCommonsClientHttpRequestFactory ( String user,
                                                        String password )
    {
        this.user     = user;
        this.password = password;
    }

    @Override
    public HttpClient getHttpClient ()
    {
        HttpClient client = super.getHttpClient();

        if ( this.user != null )
        {
            client.getState().setCredentials(
                AuthScope.ANY,
                new UsernamePasswordCredentials( this.user, this.password ));
        }

        return client;
    }
}

<bean class="org.springframework.web.client.RestTemplate">
    <property name  = "requestFactory">
        <bean class = "ArtifactoryCommonsClientHttpRequestFactory">
            <constructor-arg name="user"     value="..."/>
            <constructor-arg name="password" value="..."/>
        </bean>
    </property>

    ...
</bean>

Artifactory encrypted password can be specified as a "password".

But this approach didn’t work well in all scenarios, particularly in DELETE calls which went out unauthorized so I switched to setting an "Authorization" HTTP header, as shown in “Working with a RestTemplate” section above.

9. Related links.

• org.springframework.web.client.RestTemplate
• Spring Framework Reference Documentation:
  Accessing RESTful services on the Client
• SpringSource Team Blog:
  REST in Spring 3: RestTemplate
• Accessing RESTful services configured with SSL using RestTemplate
• Sending credentials (user/pass) with RestTemplates – doesn’t work for me ?
• REST API returns 406 (Not Acceptable) when working with Spring 3 RestTemplate
• SPR-7357:
  Provide a way to read an InputStream with RestTemplate
• RTFACT-3328:
  Artifactory REST API returns content type “application/vnd.org.jfrog.artifactory.storage.folderinfo+json” instead of “application/json”
• RTFACT-3385:
  Unable to download file of size 2.36 GB from Artifactory
• SPR-7331:
  Provide a “username” and “password” properties for RestTemplate
• SPR-6719:
  CommonsClientHttpRequestFactory getHttpClient() returns HttpClient from Commons HttpClient 3.x which has been EOL’d
• WGET for Windows

10. Summary.

Overall, it took some time to implement ArtifactoryDAO and sort out all issues involved: Spring converters, request and response headers, authentication, downloading large files, etc .. But it now works extremely well, response time of each REST invocation is normally below 10 ms and Artifactory Hudson plugin eats 2.5Gb files for breakfast!