Sounds like it ought to have worked. Maybe sprinkle the sugar powder on a bit earlier, while it's still a bit wet? It needs to get into the surface so that when it is washed out, microscopic little pits are left which gives the matt surface. All you can do is put a little more silicone over it and try again. It's mostly worked for me, though I do sometimes get a shiny patch where the powder doesn't seem to have made an impression.
Probably not enough powder. I generally coat the silicone generously, and it looks very powdery indeed, but is fine after the silicone has set and is washed.
And the other possibility is that the silicone had already largely set before the powder was put on.