Failed to create CUPTI subcriber.

I get the following error, generated by using nvprof/nvvp with tensorflow

[Error]Internal: Failed to create CUPTI subcriber

I just search the google and find out that this error message comes from tensorflow. I think tensorflow already uses cupti to trace itself, as nvprof cannot profile the cupti program because nvprof also uses cupti, this is excepted.

Status GPUTracerImpl::Start() {
VLOG(1) << “GPUTracer::Start”;
mutex_lock l(mu_);
if (enabled_) {
return errors::FailedPrecondition(“GPUTracer is already enabled.”);
}
// There can only be one CUPTI subscriber. If we can’t create one then
// there is another trace in progress (possibly by external code).
CUptiResult ret;
ret = cupti_wrapper_->Subscribe(&subscriber_, (CUpti_CallbackFunc)ApiCallback,
this);
if (ret == CUPTI_ERROR_MAX_LIMIT_REACHED) {
return errors::Unavailable(“CUPTI subcriber limit reached.”);
} else if (ret != CUPTI_SUCCESS) {
return errors::Internal(“Failed to create CUPTI subcriber.”);
}

// Register as a TraceEngine to receive ScopedAnnotations.
port::Tracing::RegisterEngine(this);

Is there a workaround for this?

Will check this with dev.

For the “Failed to create CUPTI subscriber error”, here is the explanation from dev:

It’s not allowed to have multiple GPU profilers operating at the same time. Visual Profiler and nvprof are not intended to be used with applications that make use of the CUPTI.

This is documented in the profiler documentation but this is covered under events and metrics limitations:
Here are a couple of reasons why Visual Profiler may fail to gather metric or event information. More than one tool is trying to access the GPU. To fix this issue please make sure only one tool is using the GPU at any given point. Tools include the CUDA command line profiler, Parallel NSight Analysis Tools and Graphics Tools, and applications that use either CUPTI or PerfKit API (NVPM) to read event values.