Contact Support

S3 Object Operations

Just like bucket operations, certain operations can be performed on/with objects as well.

All requests mentioned here require authentication.

Object URLs

Aside from list operations, all operations operate on the object URL, which follows the format of the Bucket URL and object name (i.e. <Bucket URL>/<object name>).

List Objects

Objects in a bucket can be listed by performing a GET request on the Bucket URL.

The below lists the objects in bucket example. (The delimiter parameter controls "path" parsing. %2F is the URL encoding for /.)

# BEGIN REQUEST #
GET /?delimiter=%2F HTTP/1.1
Host: example.us-east1.s3.netfire.com
Content-Length: 0
X-AMZ-Date: 20230914T000124Z
X-AMZ-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=88D7KRTO4HXGERCSE4TV/20230914/us-east/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=f203ccfddc4cf82d327360c122dd5449dbe65a76a8a9f8bd4b9223928afc2ff5
# END REQUEST #

# BEGIN RESPONSE #
HTTP/1.1 200 OK
Transfer-Encoding: chunked
X-AMZ-Request-ID: tx0000072fe7fdb79000625-0065024d54-978656-us-east1
Content-Type: application/xml
Date: Thu, 14 Sep 2023 00:01:24 GMT
Connection: close

## BEGIN RESPONSE BODY ##
<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Tenant>NetFire</Tenant><Name>example</Name><Prefix></Prefix><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><CommonPrefixes><Prefix>exampleprefix1/</Prefix></CommonPrefixes><Contents><Key>example.txt</Key><LastModified>2023-03-06T17:34:12.650Z</LastModified><ETag>&quot;ce90a5f32052ebbcd3b20b315556e154&quot;</ETag><Size>46</Size><StorageClass>STANDARD</StorageClass><Owner><ID>NetFire$example</ID><DisplayName>Example</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>meme.png</Key><LastModified>2023-09-13T23:52:48.529Z</LastModified><ETag>&quot;eb1c50662e6a9f64f55386584e483bc5&quot;</ETag><Size>990237</Size><StorageClass>STANDARD</StorageClass><Owner><ID>NetFire$example</ID><DisplayName>Example</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>testfile</Key><LastModified>2023-03-13T11:04:49.657Z</LastModified><ETag>&quot;18f2ae724cf04ac1b5306f8cf9a6137d&quot;</ETag><Size>32</Size><StorageClass>STANDARD</StorageClass><Owner><ID>NetFire$example</ID><DisplayName>Example</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>testfile.alt</Key><LastModified>2023-03-07T03:14:33.927Z</LastModified><ETag>&quot;e9409172a4036cc688f169c72131e921&quot;</ETag><Size>9</Size><StorageClass>STANDARD</StorageClass><Owner><ID>NetFire$example</ID><DisplayName>Example</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>upfile</Key><LastModified>2023-03-07T05:38:16.622Z</LastModified><ETag>&quot;909ef0d762160c163d5bf71adca22ea8&quot;</ETag><Size>17</Size><StorageClass>STANDARD</StorageClass><Owner><ID>NetFire$example</ID><DisplayName>Example</DisplayName></Owner><Type>Normal</Type></Contents><Marker></Marker></ListBucketResult>
## END RESPONSE BODY ##
# END RESPONSE #

"Prettified" response body:

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Tenant>NetFire</Tenant>
  <Name>example</Name>
  <Prefix/>
  <MaxKeys>1000</MaxKeys>
  <Delimiter>/</Delimiter>
  <IsTruncated>false</IsTruncated>
  <CommonPrefixes>
    <Prefix>exampleprefix1/</Prefix>
  </CommonPrefixes>
  <Contents>
    <Key>example.txt</Key>
    <LastModified>2023-03-06T17:34:12.650Z</LastModified>
    <ETag>"ce90a5f32052ebbcd3b20b315556e154"</ETag>
    <Size>46</Size>
    <StorageClass>STANDARD</StorageClass>
    <Owner>
      <ID>NetFire$example</ID>
      <DisplayName>Example</DisplayName>
    </Owner>
    <Type>Normal</Type>
  </Contents>
  <Contents>
    <Key>meme.png</Key>
    <LastModified>2023-09-13T23:52:48.529Z</LastModified>
    <ETag>"eb1c50662e6a9f64f55386584e483bc5"</ETag>
    <Size>990237</Size>
    <StorageClass>STANDARD</StorageClass>
    <Owner>
      <ID>NetFire$example</ID>
      <DisplayName>Example</DisplayName>
    </Owner>
    <Type>Normal</Type>
  </Contents>
  <Contents>
    <Key>testfile</Key>
    <LastModified>2023-03-13T11:04:49.657Z</LastModified>
    <ETag>"18f2ae724cf04ac1b5306f8cf9a6137d"</ETag>
    <Size>32</Size>
    <StorageClass>STANDARD</StorageClass>
    <Owner>
      <ID>NetFire$example</ID>
      <DisplayName>Example</DisplayName>
    </Owner>
    <Type>Normal</Type>
  </Contents>
  <Contents>
    <Key>testfile.alt</Key>
    <LastModified>2023-03-07T03:14:33.927Z</LastModified>
    <ETag>"e9409172a4036cc688f169c72131e921"</ETag>
    <Size>9</Size>
    <StorageClass>STANDARD</StorageClass>
    <Owner>
      <ID>NetFire$example</ID>
      <DisplayName>Example</DisplayName>
    </Owner>
    <Type>Normal</Type>
  </Contents>
  <Contents>
    <Key>upfile</Key>
    <LastModified>2023-03-07T05:38:16.622Z</LastModified>
    <ETag>"909ef0d762160c163d5bf71adca22ea8"</ETag>
    <Size>17</Size>
    <StorageClass>STANDARD</StorageClass>
    <Owner>
      <ID>NetFire$example</ID>
      <DisplayName>Example</DisplayName>
    </Owner>
    <Type>Normal</Type>
  </Contents>
  <Marker/>
</ListBucketResult>

For reference's sake, if delimiter was not provided, this is an example of the same request path:

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Tenant>NetFire</Tenant>
  <Name>example</Name>
  <Prefix/>
  <MaxKeys>1000</MaxKeys>
  <IsTruncated>false</IsTruncated>
  <Contents>
    <Key>example.txt</Key>
    <LastModified>2023-03-06T17:34:12.650Z</LastModified>
    <ETag>"ce90a5f32052ebbcd3b20b315556e154"</ETag>
    <Size>46</Size>
    <StorageClass>STANDARD</StorageClass>
    <Owner>
      <ID>NetFire$example</ID>
      <DisplayName>Example</DisplayName>
    </Owner>
    <Type>Normal</Type>
  </Contents>
  <Contents>
    <Key>exampleprefix1/exampleprefix2/main.py</Key>
    <LastModified>2023-03-08T22:09:20.737Z</LastModified>
    <ETag>"fe08890fb28a0fa6fd48b988a46c5f9c"</ETag>
    <Size>9313</Size>
    <StorageClass>STANDARD</StorageClass>
    <Owner>
      <ID>NetFire$example</ID>
      <DisplayName>Example</DisplayName>
    </Owner>
    <Type>Normal</Type>
  </Contents>
  <Contents>
    <Key>meme.png</Key>
    <LastModified>2023-09-13T23:52:48.529Z</LastModified>
    <ETag>"eb1c50662e6a9f64f55386584e483bc5"</ETag>
    <Size>990237</Size>
    <StorageClass>STANDARD</StorageClass>
    <Owner>
      <ID>NetFire$example</ID>
      <DisplayName>Example</DisplayName>
    </Owner>
    <Type>Normal</Type>
  </Contents>
  <Contents>
    <Key>testfile</Key>
    <LastModified>2023-03-13T11:04:49.657Z</LastModified>
    <ETag>"18f2ae724cf04ac1b5306f8cf9a6137d"</ETag>
    <Size>32</Size>
    <StorageClass>STANDARD</StorageClass>
    <Owner>
      <ID>NetFire$example</ID>
      <DisplayName>Example</DisplayName>
    </Owner>
    <Type>Normal</Type>
  </Contents>
  <Contents>
    <Key>testfile.alt</Key>
    <LastModified>2023-03-07T03:14:33.927Z</LastModified>
    <ETag>"e9409172a4036cc688f169c72131e921"</ETag>
    <Size>9</Size>
    <StorageClass>STANDARD</StorageClass>
    <Owner>
      <ID>NetFire$example</ID>
      <DisplayName>Example</DisplayName>
    </Owner>
    <Type>Normal</Type>
  </Contents>
  <Contents>
    <Key>upfile</Key>
    <LastModified>2023-03-07T05:38:16.622Z</LastModified>
    <ETag>"909ef0d762160c163d5bf71adca22ea8"</ETag>
    <Size>17</Size>
    <StorageClass>STANDARD</StorageClass>
    <Owner>
      <ID>NetFire$example</ID>
      <DisplayName>Example</DisplayName>
    </Owner>
    <Type>Normal</Type>
  </Contents>
  <Marker/>
</ListBucketResult>

Note that not only is the /ListBucketResult/Delimiter element missing if not specified, but also the difference in behavior. With delimiter specified, only the "first level" of objects are displayed and the /ListBucketResult/CommonPrefixes acts as a list of "directories". Without delimiter, all objects are displayed -- in this case, including exampleprefix1/exampleprefix2/main.py.

Prefixes, Not Paths

At first glance this may seem that there actually are directories in object storage! But no; as you'll recall, this is actually an object "prefix" -- merely a text string. This allows for some clever tricks if you have public buckets as it can ease transition from more traditional filesystem-backed webservers that may have files in certain directories.

📘

Transitioning to NetFire Cloud Storage can be painless!

If a prefix is used to match the same path of a file/files in your current web application, a transition from your webserver to NetFire Cloud Storage as a CDN can be seamless. Get in touch and our staff will help you make the jump!

For clarification, prefixes are more like a filename than an actual directory path/folder path. An object is "named" foo/bar/baz.tzt, it isn't a file named baz.txt in a subdirectory of foo/ called bar/.

Create/Update an Object ("Upload a File")

To upload a file to a bucket (technically you're "creating an object" in the bucket and populating that object's storage with bytes, or replacing the bytes if the object name already exists), a PUT request is performed via the Object URL.

📘

If using a bucket with versioning enabled, you can have multiple versions of an object!

Unlike (most) of more traditional block filesystems, objects can be versioned. This can cause unexpected usage towards quota if you aren't prepared to account for this, but can provide useful archive/reference features.

# BEGIN REQUEST #
PUT /cat.png HTTP/1.1
Host: example.us-east1.s3.netfire.com
Content-Length: 24351
Authorization: AWS4-HMAC-SHA256 Credential=88D7KRTO4HXGERCSE4TV/20230914/us-east/s3/aws4_request,SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class,Signature=54d0c09adee1e1ef039d95a5f8e75fdde827f2e846078bb444a4576e8bc56f95
Content-Type: image/png
X-AMZ-Content-SHA256: b8b129955ac65ac5f6a9bfa9d5fd8a05ba5a0e50afd8aa0edd062f7d0f16fb73
X-AMZ-Date: 20230308T223150Z
X-AMZ-Meta-s3cmd-attrs: atime:1678314660/ctime:1678314670/gid:1000/gname:mylocalgroup/md5:8783c70360231686325c4952c87e5bc4/mode:33188/mtime:1678314660/uid:1000/uname:mylocaluser
X-AMZ-Storage-Class: STANDARD

## BINARY DATA BODY EXPUNGED ##

# END REQUEST #

# BEGIN RESPONSE #
HTTP/1.1 200 OK
Content-Length: 0
ETag: "8783c70360231686325c4952c87e5bc4"
Accept-Ranges: bytes
X-AMZ-Request-ID: tx00000bb6658690ad616c3-0065024fb0-978656-us-east1
Date: Thu, 14 Sep 2023 00:11:28 GMT
Connection: close
# END RESPONSE #

As noticed, the actual bytes that make up a PNG of a cat for the HTTP request body have been removed (mostly for brevity), but the file cat.png has now been created as an object /cat.png in the bucket example. If one wanted to serve this file from a website, either the bucket itself could be made public via a bucket policy or only this object could be made public and it can be anonymously accessible (meaning clients wouldn't need to authenticate to download it) as https://example.us-east1.netfire.s3.netfire.net/cat.png without exposing other objects in the bucket.

Also worth taking special notice is the X-AMZ-Meta-s3cmd-attrs header. X-AMZ-Meta-* is an arbitrary header prefix that lets you define additional attributes about this object that can be retrieved via a HEAD request. In s3cmd's case, it stores information about the file so it can restore ownership, permissions, and other OS/filesystem attributes.

Get an Object ("Download a File")

An object can be downloaded by performing a GET request against an Object URL, which will retrieve (download) an object (file).

🚧

Unauthenticated GETs require path-based references!

DNS-based bucketing will not work for unauthenticated requests. If you are linking to a file to be used without authentication (e.g. hosting an image on your website), the URL/path-based bucket reference must be used.

The alternative is to use a presigned URL.

# BEGIN REQUEST #
GET /cat.png HTTP/1.1
Host: example.us-east1.s3.netfire.com
Authorization: AWS4-HMAC-SHA256 Credential=88D7KRTO4HXGERCSE4TV/20230914/us-east/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=03ec128801e06bc5e11b9ae61f7d156543d0018dbc968836bd9ee5103880b468
X-AMZ-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-AMZ-Date: 20230914T001635Z
# END REQUEST #

# BEGIN RESPONSE #
HTTP/1.1 200 OK
Content-Length: 24351
Accept-Ranges: bytes
Last-Modified: Thu, 14 Sep 2023 00:11:28 GMT
X-RGW-Object-Type: Normal
ETag: "8783c70360231686325c4952c87e5bc4"
X-AMZ-Meta-s3cmd-attrs: atime:1694649012/ctime:1694649006/gid:1000/gname:mylocalgroup/md5:8783c70360231686325c4952c87e5bc4/mode:33188/mtime:1678332713/uid:1000/uname:mylocaluser
X-AMZ-Storage-Class: STANDARD
X-AMZ-Request-ID: tx00000334b0c6ebb0f1cff-00650250e3-978656-us-east1
Content-Type: image/png
Date: Wed, 14 Sep 2023 00:16:35 GMT
Connection: close

## BINARY DATA BODY EXPUNGED ##

# END RESPONSE #

Delete an Object ("Delete a File")

An object can be removed from a bucket by issuing a DELETE request on the Object URL. The object testfile.alt is removed from the example bucket below.

# BEGIN REQUEST #
DELETE /testfile.alt HTTP/1.1
Host: example.us-east1.s3.netfire.com
Authorization: AWS4-HMAC-SHA256 Credential=88D7KRTO4HXGERCSE4TV/20230930/us-east/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=eeb028db62b8d4c3ccd1ee817ea549b76b33eab70aa22190d094479c04dc5743
X-AMZ-Content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-AMZ-Date: 20230930T103840Z
# END REQUEST #

# BEGIN RESPONSE #
HTTP/1.1 204 No Content
X-AMZ-Request-ID: tx000003174adcf00abe0dc-006517fab1-9649c0-us-east1
Date: Sat, 30 Sep 2023 10:38:41 GMT
Connection: close
# END RESPONSE #