# Storage

The **MEMO Storage Middleware** provides developers, enterprises, and individual users with an **extensible, private, reliable, and composable storage platform**. Users can flexibly choose their preferred underlying storage system—currently supporting **decentralized storage (IPFS)** and **distributed storage (MEFS)**. Additionally, MEMO offers a **file marketplace** where users can freely trade access permissions to their files and retrieve files via keyword-based searches.

The middleware service runs by default on port **8080**.

## Authentication

The middleware uses [**DID**](https://www.w3.org/TR/did-core/) **(Decentralized Identifier)** for login, enabling verifiable and decentralized digital identity. Specifically, we use [**Memo DID**](https://github.com/memoio/did-docs/blob/master/memo-did-design.md) as the account system. Users must first create a **Memo DID** and add verification methods before logging in. For details, refer to:

* [did-go-sdk](https://github.com/memoio/did-go-sdk)
* [did-js-sdk](https://github.com/memoio/did-js-sdk)

### 1.1 Login

Logging in establishes agreement on a **random token** (generated by the user) and resets the **requestID** to **0**. Subsequent restricted API calls require signing the token and an incrementing requestID. Login must be completed within **1 minute** of signing.

Endpoint:

`POST http://localhost:8080/login`

Headers:

None

Request Parameters:

| Parameter | Variable      | Type   | Required | Description                        |
| --------- | ------------- | ------ | -------- | ---------------------------------- |
| did       | MEMO DID      | String | Yes      | Login ID                           |
| token     | Random string | String | Yes      | User-generated random string (hex) |
| timestamp | Timestamp     | int64  | Yes      | Timestamp at signing               |
| signature | Signature     | string | Yes      | Signature of the above parameters  |

Response: "Login success!"

Example Request (Go):

```go
package main

import (
	"bytes"
	"crypto/rand"
	"encoding/binary"
	"encoding/json"
	"flag"
	"io/ioutil"
	"log"
	"net/http"
	"time"

	"github.com/ethereum/go-ethereum/common/hexutil"
	"github.com/ethereum/go-ethereum/crypto"
)

func main() {
	secretKey := flag.String("sk", "", "the sk to signature")
	did := flag.String("did", "did:memo:d687daa192ffa26373395872191e8502cc41fbfbf27dc07d3da3a35de57c2d96", "the did to login")
    token := []byte("memo.test")

	req, err := GetLoginRequest(*did, token, *secretKey)
	if err != nil {
		panic(err.Error())
	}

	client := &http.Client{Timeout: time.Minute}
	res, err := client.Do(req)
	if err != nil {
		panic(err.Error())
	}
	defer res.Body.Close()

	body, err := ioutil.ReadAll(res.Body)
	if err != nil {
		panic(err.Error())
	}

	if res.StatusCode != http.StatusOK {
		log.Printf("respond code[%d]: %s\n", res.StatusCode, string(body))
	} else {
		log.Println(string(body))
	}

}

func GetLoginRequest(did string, token []byte, privateKeyHex string) (*http.Request, error) {
    var url = "http://localhost:8080/login"
	var timestampBuf = make([]byte, 8)
	var timestamp = time.Now().Unix()

	privateKey, err := crypto.HexToECDSA(privateKeyHex)
	if err != nil {
		return nil, err
	}

	binary.LittleEndian.PutUint64(timestampBuf, uint64(timestamp))
	hash := crypto.Keccak256([]byte(did), token, timestampBuf)
	signature, err := crypto.Sign(hash, privateKey)
	if err != nil {
		return nil, err
	}

	var payload = make(map[string]interface{})
	payload["did"] = did
	payload["token"] = hexutil.Encode(token)
	payload["timestamp"] = timestamp
	payload["signature"] = hexutil.Encode(signature)

	b, err := json.Marshal(payload)
	if err != nil {
		return nil, err
	}

	req, err := http.NewRequest("POST", url, bytes.NewReader(b))

	return req, err
}
```

Error Codes:

| HTTP Code | Error          | Description                                         |
| --------- | -------------- | --------------------------------------------------- |
| 401       | Authentication | Missing paramenter;                                 |
| 500       | InternalError  | We encountered an internal error, please try again. |

### 1.2 Get Login Info

Retrieve login details (**token, requestID**, and last login timestamp) after successful authentication.

Endpoint:

`GET http://localhost:8080/login?did={did}`

Headers:

None

Response Parameters:

| Parameter | Variable      | Type   | Description                          |
| --------- | ------------- | ------ | ------------------------------------ |
| token     | Random string | String | User-generated token (hex)           |
| timestamp | Timestamp     | int64  | Timestamp at signing                 |
| requestID | Request ID    | string | Incremented for restricted API calls |

### 1.3 Access Restricted APIs

After login, restricted APIs require signed requests with the following additional parameters:

| Parameter | Variable      | Type   | Required | Description                |
| --------- | ------------- | ------ | -------- | -------------------------- |
| did       | MEMO DID      | String | Yes      | Login ID                   |
| token     | Random string | String | Yes      | User-generated token (hex) |
| requestID | Request ID    | int64  | Yes      | Incremented per call       |
| payload   | File hash     | string | No       | Required for file uploads  |
| signature | Signature     | string | Yes      | Signature of the above     |

Example (File List Query in Go):

```go
package main

import (
	"bytes"
	"encoding/binary"
	"encoding/json"
	"flag"
	"io/ioutil"
	"log"
	"net/http"
	"time"

	"github.com/ethereum/go-ethereum/common/hexutil"
	"github.com/ethereum/go-ethereum/crypto"
)

func main() {
	secretKey := flag.String("sk", "", "the sk to signature")
	did := flag.String("did", "did:memo:d687daa192ffa26373395872191e8502cc41fbfbf27dc07d3da3a35de57c2d96", "the did to login")
	token := []byte("memo.test")

	for index := 0; index < 5; index++ {
		req, err := GetVerifyIdentityRequest(*did, token, int64(index), *secretKey)
		if err != nil {
			panic(err.Error())
		}

		client := &http.Client{Timeout: time.Minute}
		res, err := client.Do(req)
		if err != nil {
			panic(err.Error())
		}
		defer res.Body.Close()

		body, err := ioutil.ReadAll(res.Body)
		if err != nil {
			panic(err.Error())
		}

		if res.StatusCode != http.StatusOK {
			log.Printf("respond code[%d]: %s\n", res.StatusCode, string(body))
		} else {
			log.Println(string(body))
		}
	}

}

func GetVerifyIdentityRequest(did string, token []byte, requestID int64, privateKeyHex string) (*http.Request, error) {
	var requestIDBuf = make([]byte, 8)
	var url = "http://localhost:8080/mefs/listobjects"

	privateKey, err := crypto.HexToECDSA(privateKeyHex)
	if err != nil {
		return nil, err
	}

	binary.LittleEndian.PutUint64(requestIDBuf, uint64(requestID))
	hash := crypto.Keccak256([]byte(did), token, requestIDBuf)
	signature, err := crypto.Sign(hash, privateKey)
	if err != nil {
		return nil, err
	}

	var payload = make(map[string]interface{})
	payload["did"] = did
	payload["token"] = hexutil.Encode(token)
	payload["requestID"] = requestID
	payload["signature"] = hexutil.Encode(signature)

	b, err := json.Marshal(payload)
	if err != nil {
		return nil, err
	}

	req, err := http.NewRequest("GET", url, bytes.NewReader(b))

	return req, err
}

```

## Traffic Voucher Hash

Get a **traffic voucher hash** for signed uploads.

Endpoint:

`GET http://localhost:8080/traffichash`

Headers:

* Content-Type: multipart/form-data
* Authorization: "Bearer {access\_token}"

Request Parameters:

| Parameter | Variable     | Variable | Required | Description     |
| --------- | ------------ | -------- | -------- | --------------- |
| size      | Voucher size | String   | Yes      | Cumulative size |

Response (JSON):

```json
{ "hash": "voucher_hash_string" }

```

## File Upload

Upload files after login. Files are stored in a **bucket matching the user’s DID**. If the bucket doesn’t exist, it’s created automatically.

**Storage Limits:**

* Users must have sufficient **paid/free storage quota** (default: 3-year validity).
* Admins may grant free space (e.g., via Twitter promotions).

Endpoints:

* MEFS: `POST http://localhost:8080/mefs/`
* IPFS: `POST http://localhost:8080/ipfs/`

Headers:

* Content-Type: multipart/form-data
* Authorization: "Bearer {access\_token}"

Request Parameters:

| Parameter | Variable          | Variable | Required | Description         |
| --------- | ----------------- | -------- | -------- | ------------------- |
| file      | File to upload    | File     | Yes      | File data           |
| size      | Voucher size      | String   | Yes      | Cumulative size     |
| sign      | Voucher signature | String   | Yes      | Signed voucher hash |

Response(JSON)：

```json
{ "cid": "bafybeie2ph7iokrckc5iy6xu7npa4xqst6ez5ewb7j2igeilxjaw6sd2qi" }
```

Error Codes:

| HTTP Code | Error          | Description                                          |
| --------- | -------------- | ---------------------------------------------------- |
| 401       | Authentication | Token is Null; Invalid token payload; Invalid token; |
| 500       | InternalError  | We encountered an internal error, please try again.  |
| 518       | Storage        | storage not support                                  |

&#x20;

## File Download

Users can download files using the file's `etag` (CID).

**Endpoint:**

* MEFS: `http://{ip}:{port}/mefs/{cid}`
* IPFS: `http://{ip}:{port}/ipfs/{cid}`

**Examples:**

* Download from MEFS:\
  `http://localhost:8081/mefs/bafkreifzwcj6vkozz6brwutpxl3hqneran4y5vtvirnbrtw3l2m3jtlgq4`
* Download from IPFS:\
  `http://localhost:8081/ipfs/bafkreifzwcj6vkozz6brwutpxl3hqneran4y5vtvirnbrtw3l2m3jtlgq4`

**Method:**\
`GET`

**Headers:**\
None

**Request Parameters:**

| Parameter | Variable          | Type     | Required | Description         |
| --------- | ----------------- | -------- | -------- | ------------------- |
| `size`    | Voucher size      | `String` | Yes      | Cumulative size     |
| `sign`    | Voucher signature | `String` | Yes      | Signed voucher hash |

**Response (`DataFromReader`):**\
Returns the file stream.

| Parameter       | Description                    | Value |
| --------------- | ------------------------------ | ----- |
| `code`          | HTTP status code               | `200` |
| `contentLength` | File size (bytes)              |       |
| `contentType`   | File MIME type                 |       |
| `reader`        | `io.Reader` (file data stream) |       |

**Error Codes:**

| HTTP Code | Error         | Description                              |
| --------- | ------------- | ---------------------------------------- |
| `500`     | InternalError | Internal server error, please try again. |
| `518`     | Storage       | Storage type not supported.              |
| `517`     | Address       | Address parameter is missing or invalid. |

***

*(Note: Replace `{ip}`, `{port}`, and `{cid}` placeholders with actual values in requests.)*

## File Deletion

Deletes uploaded files. Only supports deletion for MEFS storage type.

**Endpoint:** `http://{ip}:{port}/mefs/delete`

**Method:** `GET`

**Headers:**

| Header        | Variable               | Type   | Required | Description                                                                           |
| ------------- | ---------------------- | ------ | -------- | ------------------------------------------------------------------------------------- |
| Authorization | "Bearer {accessToken}" | string | Yes      | Access token generated during login. If expired, refresh to obtain a new valid token. |

**Request Parameters:**

| Parameter | Variable | Type | Required | Description                        |
| --------- | -------- | ---- | -------- | ---------------------------------- |
| id        | File ID  | id   | Yes      | The file ID (index from file list) |

**Response (JSON):**

| Parameter | Type   | Description            |
| --------- | ------ | ---------------------- |
| Status    | string | "Success" or "Failure" |

**Example Request:**

```bash
curl -X GET "http://localhost:8080/mefs/delete?id=123" \
-H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
```

**Notes:**

* This operation is only available for files stored in MEFS
* The file ID can be obtained from the file list API response
* Successful deletion returns HTTP 200 with status "Success"
* Failed deletion returns appropriate error code with status "Failure"

**Error Conditions:**

* Invalid or expired access token (401 Unauthorized)
* File not found (404 Not Found)
* Permission denied (403 Forbidden)
* Internal server error (500 Internal Server Error)

Sure! Here's the English translation of the text:

***

## fileDNS

The middleware is based on [Mfile DID](https://github.com/memoio/did-docs/blob/master/mfile-did-design.md) and enables permission control for files, allowing users to manage their file access rights independently. Files associated with Mfile DIDs will be listed on fileDNS, enabling other users to purchase usage rights. This mechanism allows creators to profit from their files through Mfile DID.

### Upload to fileDNS

No HTTP API is provided. Uploading to fileDNS can be done via the [did-js-sdk](https://github.com/memoio/js-did) or [did-go-sdk](https://github.com/memoio/go-did).

### Grant Read Permissions

No HTTP API is provided. Read permissions can be granted via the [did-js-sdk](https://github.com/memoio/js-did) or [did-go-sdk](https://github.com/memoio/go-did).

### Purchase Read Permissions

No HTTP API is provided. Read permissions can be purchased via the [did-js-sdk](https://github.com/memoio/js-did) or [did-go-sdk](https://github.com/memoio/go-did).

### Modify File Information in DNS

No HTTP API is provided. File information in DNS can be modified using either the `did-js-sdk` or `did-go-sdk`, including:

* **Change File Owner**: Use [did-js-sdk](https://github.com/memoio/js-did) or [did-go-sdk](https://github.com/memoio/go-did) to change file ownership.
* **Change File Type** (0 for private, 1 for public): Use [did-js-sdk](https://github.com/memoio/js-did) or [did-go-sdk](https://github.com/memoio/go-did) to update the file type.
* **Change File Price** (unit: attomemo): Use [did-js-sdk](https://github.com/memoio/js-did) or [did-go-sdk](https://github.com/memoio/go-did) to modify the file price.
* **Change File Keywords**: Use [did-js-sdk](https://github.com/memoio/js-did) or [did-go-sdk](https://github.com/memoio/go-did) to update file keywords.

### Retrieve File Information from DNS

After a file is uploaded to fileDNS (which usually takes a few minutes), other users can search for the file through fileDNS. A user submits a search query, and the middleware matches it against the file name and keywords to locate related files.

**Request URL:**\
`http://localhost:8080/challenge?query={query_text}&page={page}&size={page_size}`

**Method:**\
`GET`

**Request Parameters:**\
None

**Response Parameters (Array):**

| Parameter | Type \[Length Limit] | Description                        |
| --------- | -------------------- | ---------------------------------- |
| Mid       | string               | Unique file ID                     |
| Name      | string               | File name                          |
| Size      | int64                | File size                          |
| Public    | bool                 | Whether the file is free to access |
| Price     | int64                | Price required to read the file    |
| keywords  | \[]string            | Descriptive keywords of the file   |

Here’s the English translation for the "Query Information" section:

***

## Query Information

### Query File List

Users can retrieve a list of files they have uploaded.

**Request URL:**

`http://ip:port/mefs/listobjects`

or

`http://ip:port/ipfs/listobjects`

**Request Method:** `GET`

This request returns the list of files associated with the currently logged-in account.

**Request Headers:**

| Parameter     | Variable                            | Type \[Length Limit] | Required | Description                                                              |
| ------------- | ----------------------------------- | -------------------- | -------- | ------------------------------------------------------------------------ |
| Authorization | `"Bearer <accessToken from login>"` | string               | Yes      | If expired, you can refresh the accessToken to obtain a new valid token. |

**Request Parameters:**\
None

**Response Parameters (JSON):**

| Parameter | Type \[Length Limit] | Description                            |
| --------- | -------------------- | -------------------------------------- |
| Address   | string               | Ethereum wallet address of the account |
| Storage   | string               | Indicates "mefs" or "ipfs" storage     |
| Object    | struct               | File list information                  |

Each file entry includes:

| Parameter   | Type   | Description                |
| ----------- | ------ | -------------------------- |
| ID          | int    | File index                 |
| Name        | string | File name                  |
| Size        | int64  | File size                  |
| Cid         | string | File CID                   |
| ModTime     | time   | Last modification time     |
| Public      | bool   | Whether the file is public |
| UserDefined | struct | Additional custom metadata |

**`UserDefined` Structure Includes:**

| Parameter  | Type   | Description                      |
| ---------- | ------ | -------------------------------- |
| encryption | string | File encryption method           |
| etag       | string | File ID pattern (default is CID) |

**Error Codes:**

| HTTP Status | Error Code     | Description                                         |
| ----------- | -------------- | --------------------------------------------------- |
| 401         | Authentication | Token is null; Invalid token payload; Invalid token |
| 516         | Storage        | `list object error %s`                              |
| 518         | Storage        | `storage not support`                               |

***

### Query Remaining Storage Space

Users can check their remaining storage space.

**Request URL:**

`http://ip:port/mefs/space`

or

`http://ip:port/ipfs/space`

**Request Method:** `GET`

This request will return the remaining space for the logged-in account.

**Request Headers:**

| Parameter     | Variable                            | Type \[Length Limit] | Required | Description                                                      |
| ------------- | ----------------------------------- | -------------------- | -------- | ---------------------------------------------------------------- |
| Authorization | `"Bearer <accessToken from login>"` | string               | Yes      | If expired, refresh the accessToken to obtain a new valid token. |

**Request Parameters:**\
None

**Response Parameters (JSON):**

| Parameter | Type \[Length Limit] | Description                        |
| --------- | -------------------- | ---------------------------------- |
| Nonce     | int64                | Unique nonce value                 |
| Balance   | int64                | Account balance                    |
| SizeByte  | int64                | Remaining storage space (in bytes) |
| FreeByte  | int64                | Free storage space (in bytes)      |
| Expire    | int64                | Expiration time (timestamp)        |

***

### Query Remaining Bandwidth

Users can check their remaining data traffic quota.

**Request URL:**

`http://ip:port/mefs/traffic`

or

`http://ip:port/ipfs/traffic`

**Request Method:** `GET`

This request will return the remaining bandwidth for the logged-in account.

**Request Headers:**

| Parameter     | Variable                            | Type \[Length Limit] | Required | Description                                                      |
| ------------- | ----------------------------------- | -------------------- | -------- | ---------------------------------------------------------------- |
| Authorization | `"Bearer <accessToken from login>"` | string               | Yes      | If expired, refresh the accessToken to obtain a new valid token. |

**Request Parameters:**\
None

**Response Parameters (JSON):**

| Parameter | Type \[Length Limit] | Description                 |
| --------- | -------------------- | --------------------------- |
| Nonce     | int64                | Unique nonce value          |
| Balance   | int64                | Account balance             |
| SizeByte  | int64                | Remaining bandwidth (bytes) |
| FreeByte  | int64                | Free bandwidth (bytes)      |
| Expire    | int64                | Expiration time (timestamp) |

### Query Unit Price for Storage

The unit price for storage can be queried directly via an RPC contract call.\
Call the contract at `middleware-contracts/Proxy.sol` and use the interface `spacePrice()`, which returns a `uint64` value.

### Query Unit Price for Bandwidth

The unit price for bandwidth can be queried directly via an RPC contract call.\
Call the contract at `middleware-contracts/Proxy.sol` and use the interface `trafficPrice()`, which returns a `uint64` value.

## Purchase

### Purchase Storage

**Request URL:**

`http://ip:port/mefs/buyspace`

or

`http://ip:port/ipfs/buyspace`

**Request Method:** `GET`

This endpoint processes the storage purchase request for the logged-in account.

**Request Headers:**

| Parameter     | Variable                            | Type \[Length Limit] | Required | Description                                                      |
| ------------- | ----------------------------------- | -------------------- | -------- | ---------------------------------------------------------------- |
| Authorization | `"Bearer <accessToken from login>"` | string               | Yes      | If expired, refresh the accessToken to obtain a new valid token. |

**Request Parameters:**\
None

**Response Parameters (JSON):**

| Parameter | Type   | Description      |
| --------- | ------ | ---------------- |
| tx        | string | Transaction hash |

### Purchase Bandwidth

**Request URL:**

`http://ip:port/mefs/buytraffic`

or

`http://ip:port/ipfs/buytraffic`

**Request Method:** `GET`

This endpoint processes the bandwidth purchase request for the logged-in account.

**Request Headers:**

| Parameter     | Variable                            | Type \[Length Limit] | Required | Description                                                      |
| ------------- | ----------------------------------- | -------------------- | -------- | ---------------------------------------------------------------- |
| Authorization | `"Bearer <accessToken from login>"` | string               | Yes      | If expired, refresh the accessToken to obtain a new valid token. |

**Request Parameters:**\
None

**Response Parameters (JSON):**

| Parameter | Type   | Description      |
| --------- | ------ | ---------------- |
| tx        | string | Transaction hash |

### Apply for Free Storage

Users can apply for free storage from the StorMark admin, for example, by posting a tweet with specified content.\
Then, the admin manually issues the free storage using a CLI tool.\
(Once the admin panel is implemented, this can also be done via a browser interface.)

Current CLI tool codebase: `https://github.com/memoio/middleware-contracts`\
To build and install the CLI tool:

```bash
go install
```

To issue free storage via CLI:

```bash
middleware-contracts admin free --type=space
```

Refer to the command help for specific options and parameters.

### Apply for Free Bandwidth

Users can apply for free bandwidth from the StorMark admin, for example, by posting a tweet with specified content.\
Then, the admin manually issues the free bandwidth using the CLI tool.\
(Once the admin panel is implemented, this can also be done via a browser interface.)

Current CLI tool codebase: `https://github.com/memoio/middleware-contracts`\
To build and install the CLI tool:

```bash
go install
```

To issue free bandwidth via CLI:

```bash
middleware-contracts admin free --type=traffic
```

Refer to the command help for specific options and parameters.