Pure elemental silver is not believed to be toxic (though you probably couldn't afford to eat enough of it to know for sure). Many compounds made with silver are toxic. Argyria, for instance, is often caused by consuming colloidal silver.