<link href="../style_sheet.css" rel="stylesheet" type="text/css">

# DMA

## API

### From main memory to local store

    EA_FORCE_INLINE void DmaGet(void* dst, const void* src, size_t size, uint32_t tag = JOB_UTIL_DMA_TAG_DEFAULT);

Copy size bytes of data from the effective address `src` in main memory to the
local address `dst`.  Both `src` and `dst` addresses must be 16 byte aligned.
Size must be a multiple of 16 bytes.  Addresses must point to valid allocated
memory.  (Don't just round up the size to the next multiple of 16 bytes.)

The DMA tag can be used for debugging (see troubleshooting below)

    template<class T> EA_FORCE_INLINE T DmaGet(const T* src, uint32_t tag = JOB_UTIL_DMA_TAG_DEFAULT);

Return a value based on it's type from the effective address `src` in main
memory.  This is restricted to types that are 1, 2, 4, 8, or 16 bytes in size,
and aligned to their size.  Larger types should aligned properly and  Dma'd
using the non templated DmaGet from above to avoid having to copy the result
from a temporary buffer.

### From local store to main memory

    EA_FORCE_INLINE void DmaPut(void* dst, const void* src, size_t size, uint32_t tag = JOB_UTIL_DMA_TAG_DEFAULT);

Does the reverse of DmaGet with all the same limitations.

    template<class T> EA_FORCE_INLINE void DmaPut(T value, T* dst, uint32_t tag = JOB_UTIL_DMA_TAG_DEFAULT);

Does the reverse of DmaGet<T> with all the same limitations.

### Asynchronous DMA

Before using asynchronous DMA I highly recommend getting things working with
synchronous DMAs first.  If DMAs are a measurable bottle neck for your use case
try using the ForEach API, or the BatchProcessor API first before trying to
reimplement a multiply buffered system of DMAs.

    EA_FORCE_INLINE void DmaGetAsync(void* dst, const void* src, size_t size, uint32_t tag = JOB_UTIL_DMA_TAG_DEFAULT);
    EA_FORCE_INLINE void DmaPutAsync(void* dst, const void* src, size_t size, uint32_t tag = JOB_UTIL_DMA_TAG_DEFAULT);

These at exactly like the synchronous counterparts expect they return before
the transfer has completed.  The `tag` is used at wait for all DMAs assigned to
it using:

    EA_FORCE_INLINE void DmaWaitAll(uint32_t tagBitMask = JOB_UTIL_DMA_TAG_ALL);
    EA_FORCE_INLINE void DmaWaitAny(uint32_t tagBitMask = JOB_UTIL_DMA_TAG_ALL);

You can wait on multiple tags at once.  If your DMA is on tag 5 you would pass
`(1 << 5)` as the bit mask to wait on.  If you wanted to wait on tags 5 and 10
you pass ing `(1 << 5) || (1 << 10)`.

There is *no* guarantee that DMAing a region of memory happen in order.  The
DMA controller if free to bring the data in last byte first if it likes.  The
only way to be sure all memory has been transfers is to use a DmaWait call.

## Unaligned DMA

The following DMA calls perform unaligned DMA transfers:

    void DmaUnalignedGet(void * local, void const * ea, size_t size, uint32_t tag)
    void DmaUnalignedPut(void * ea, void const * local, size_t size, uint32_t tag)
    template <class T> EA_FORCE_INLINE void DmaUnalignedGet(T * local, T const * ea, uint32_t tag)
    template <class T> EA_FORCE_INLINE void DmaUnalignedPut(T * ea, T const * local, uint32_t tag)

There are no asynchronous versions, and they are slower than the aligned versions.

## PC/Xenon Emulation

If the destination and source pointers are identical then the DMA operations do
nothing.  If the addresses are different the DMA operations will copy the data
using memcpy.

[example needed]

# Troubleshooting

## Reading the crash dump

Each SPU can have up to 16 DMAs in flight simultaneously.  When a crash occurs
in an spu_thread you'll get a dump of the state of each of the 16 DMA slots.

It looks like this:

    lv2(2): MFC context:
    lv2(2):   SPUQ #0: EA 000000005c0824c0 LSA 003b0 Size 0400 Op 40 Tag 00 R/T 00
    lv2(2):            LtAddr 0000 LtSize 000 Lt/No/Q/Ea/Ce/eV/lB/Sb/Ib  --QE-V---
    lv2(2):   SPUQ #1: EA 000000005c0864c0 LSA 007b0 Size 0400 Op 40 Tag 00 R/T 00
    lv2(2):            LtAddr 0000 LtSize 000 Lt/No/Q/Ea/Ce/eV/lB/Sb/Ib  --QE-V---

Only longer (all the way down to SPUQ #f).

The important fields are:

- EA Effective Address in main memory that is being written to, or read from.

- LSA Local Store Address on the Spu that is being written to, or read from.

- Size is the number of 16-byte blocks to DMA. (So the amount transfered in
bytes is 16 times this.)

- Tag is the DMA tag.  Since you can provide this you should try to use
different tags for all your DMAs to make tracking down which DMAs are active
easier by matching up with tags are in the crash dump to the DMAs issued in
your code.

## Alignment

If you get a crash dump with either EA or LSA entries that end in something
other than a '0' then you're using addresses that are not 16 byte aligned.

Use:

    #include "EABase/EABase.h"
    
    EA_PREFIX_ALIGN(16) type EA_PREFIX_ALIGN(16) gVariableName;

To align the data.

## Minimum Size

You can DMA 4 or 8 bytes, but you have to do so using the type specific calls,
and the data has to be aligned at an address equal to its size.

The general DmaPut/DmaGet calls require at least 16 bytes of data.

## Size

General DmaPut/DmaGet calls require that amount of data to DMA be a multiple of
16 bytes.

## Data on the stack

The PS3 hardware will not DMA on-to or off=of any PPU thread's stack.  If your
EA starts with 0x4 and your are getting a crash it is likely that you've
accidental specified an address on the stack of a PPU thread.

If you are sure the function you're allocating the data in will not return until
the job has processed the data you have a few options:

- Use RAII to wrap the allocation/deallocation in an object.
- Use "static" when declaring the variable if you're *absolutely sure* the
function does not have to be reentrant. (This is the same as making the
variable global.)

It's been suggested that we should assert when a stack address is used for a
DMA.  Unfortunately it is difficult to determine if an address resides on the
stack of any thread (not just the current thread), so we are unable to provide
an assert.

## Data not in IO mapped region

Thanks to Jeff Litz for tracking down this one.  When presented with this cryptic error:

    lv2(2): exception causes:
    lv2(2):         DSI storage, MFC_DSISR 0x40000000, MFC_DAR 0x5c0824c0

He says:

    From CBE_Architecture_v101_e.pdf, page 220 (15.5 MFC Data Storage Interrupt
    Status Register (MFC_DSISR)), from Public-Cell_Doc-Apr2007-English.zip,
    (https://ps3.scedev.net/projects/hwdocs/dl/dl/1297/2451/Public-Cell_Doc-Apr2007-English.zip)...

    Bits 32-63 are the relevant bits.  MFC_DSISR 0x40000000 is bit 33, "M" ...
    PTE (Page Table Entry) is not found or a TLB miss occurs while in
    software-managed TLB mode.

    Plus, the MFC_DAR 0x5c0824c0 address looked suspiciously large for IO
    mapped memory in most projects. Having a dump of all the mapped pages would
    perfect for narrowing down these types of issues.

In this case the memory address wasn't valid.  (It was a buffer overrun.)  The fix
was to guard against accessing invalid memory.
